<!DOCTYPE html>
<!--

	Modified template for STM32CubeMX.AI purpose

	d0.1: 	jean-michel.delorme@st.com
			add ST logo and ST footer

	d2.0: 	jean-michel.delorme@st.com
			add sidenav support

	d2.1: 	jean-michel.delorme@st.com
			clean-up + optional ai_logo/ai meta data
			
==============================================================================
           "GitHub HTML5 Pandoc Template" v2.1 — by Tristano Ajmone           
==============================================================================
Copyright © Tristano Ajmone, 2017, MIT License (MIT). Project's home:

- https://github.com/tajmone/pandoc-goodies

The CSS in this template reuses source code taken from the following projects:

- GitHub Markdown CSS: Copyright © Sindre Sorhus, MIT License (MIT):
  https://github.com/sindresorhus/github-markdown-css

- Primer CSS: Copyright © 2016-2017 GitHub Inc., MIT License (MIT):
  http://primercss.io/

~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
The MIT License 

Copyright (c) Tristano Ajmone, 2017 (github.com/tajmone/pandoc-goodies)
Copyright (c) Sindre Sorhus <sindresorhus@gmail.com> (sindresorhus.com)
Copyright (c) 2017 GitHub Inc.

"GitHub Pandoc HTML5 Template" is Copyright (c) Tristano Ajmone, 2017, released
under the MIT License (MIT); it contains readaptations of substantial portions
of the following third party softwares:

(1) "GitHub Markdown CSS", Copyright (c) Sindre Sorhus, MIT License (MIT).
(2) "Primer CSS", Copyright (c) 2016 GitHub Inc., MIT License (MIT).

Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:

The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.

THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
==============================================================================-->
<html>
<head>
  <meta charset="utf-8" />
  <meta name="generator" content="pandoc" />
  <meta name="viewport" content="width=device-width, initial-scale=1.0, user-scalable=yes" />
  <meta name="keywords" content="STM32CubeMX, X-CUBE-AI, Validation flow, CLI, Code Generator" />
  <title>Evaluation report and metrics</title>
  <style type="text/css">
.markdown-body{
	-ms-text-size-adjust:100%;
	-webkit-text-size-adjust:100%;
	color:#24292e;
	font-family:-apple-system,system-ui,BlinkMacSystemFont,"Segoe UI",Helvetica,Arial,sans-serif,"Apple Color Emoji","Segoe UI Emoji","Segoe UI Symbol";
	font-size:16px;
	line-height:1.5;
	word-wrap:break-word;
	box-sizing:border-box;
	min-width:200px;
	max-width:980px;
	margin:0 auto;
	padding:45px;
	}
.markdown-body a{
	color:#0366d6;
	background-color:transparent;
	text-decoration:none;
	-webkit-text-decoration-skip:objects}
.markdown-body a:active,.markdown-body a:hover{
	outline-width:0}
.markdown-body a:hover{
	text-decoration:underline}
.markdown-body a:not([href]){
	color:inherit;text-decoration:none}
.markdown-body strong{font-weight:600}
.markdown-body h1,.markdown-body h2,.markdown-body h3,.markdown-body h4,.markdown-body h5,.markdown-body h6{
	margin-top:24px;
	margin-bottom:16px;
	font-weight:600;
	line-height:1.25}
.markdown-body h1{
	font-size:2em;
	margin:.67em 0;
	padding-bottom:.3em;
	border-bottom:1px solid #eaecef}
.markdown-body h2{
	padding-bottom:.3em;
	font-size:1.5em;
	border-bottom:1px solid #eaecef}
.markdown-body h3{font-size:1.25em}
.markdown-body h4{font-size:1em}
.markdown-body h5{font-size:.875em}
.markdown-body h6{font-size:.85em;color:#6a737d}
.markdown-body img{border-style:none}
.markdown-body svg:not(:root){
	overflow:hidden}
.markdown-body hr{
	box-sizing:content-box;
	height:.25em;
	margin:24px 0;
	padding:0;
	overflow:hidden;
	background-color:#e1e4e8;
	border:0}
.markdown-body hr::before{display:table;content:""}
.markdown-body hr::after{display:table;clear:both;content:""}
.markdown-body input{margin:0;overflow:visible;font:inherit;font-family:inherit;font-size:inherit;line-height:inherit}
.markdown-body [type=checkbox]{box-sizing:border-box;padding:0}
.markdown-body *{box-sizing:border-box}.markdown-body blockquote{margin:0}
.markdown-body ol,.markdown-body ul{padding-left:2em}
.markdown-body ol ol,.markdown-body ul ol{list-style-type:lower-roman}
.markdown-body ol ol,.markdown-body ol ul,.markdown-body ul ol,.markdown-body ul ul{margin-top:0;margin-bottom:0}
.markdown-body ol ol ol,.markdown-body ol ul ol,.markdown-body ul ol ol,.markdown-body ul ul ol{list-style-type:lower-alpha}
.markdown-body li>p{margin-top:16px}
.markdown-body li+li{margin-top:.25em}
.markdown-body dd{margin-left:0}
.markdown-body dl{padding:0}
.markdown-body dl dt{padding:0;margin-top:16px;font-size:1em;font-style:italic;font-weight:600}
.markdown-body dl dd{padding:0 16px;margin-bottom:16px}
.markdown-body code{font-family:SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace}
.markdown-body pre{font:12px SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;word-wrap:normal}
.markdown-body blockquote,.markdown-body dl,.markdown-body ol,.markdown-body p,.markdown-body pre,.markdown-body table,.markdown-body ul{margin-top:0;margin-bottom:16px}
.markdown-body blockquote{padding:0 1em;color:#6a737d;border-left:.25em solid #dfe2e5}
.markdown-body blockquote>:first-child{margin-top:0}
.markdown-body blockquote>:last-child{margin-bottom:0}
.markdown-body table{display:block;width:100%;overflow:auto;border-spacing:0;border-collapse:collapse}
.markdown-body table th{font-weight:600}
.markdown-body table td,.markdown-body table th{padding:6px 13px;border:1px solid #dfe2e5}
.markdown-body table tr{background-color:#fff;border-top:1px solid #c6cbd1}
.markdown-body table tr:nth-child(2n){background-color:#f6f8fa}
.markdown-body img{max-width:100%;box-sizing:content-box;background-color:#fff}
.markdown-body code{padding:.2em 0;margin:0;font-size:85%;background-color:rgba(27,31,35,.05);border-radius:3px}
.markdown-body code::after,.markdown-body code::before{letter-spacing:-.2em;content:"\00a0"}
.markdown-body pre>code{padding:0;margin:0;font-size:100%;word-break:normal;white-space:pre;background:0 0;border:0}
.markdown-body .highlight{margin-bottom:16px}
.markdown-body .highlight pre{margin-bottom:0;word-break:normal}
.markdown-body .highlight pre,.markdown-body pre{padding:16px;overflow:auto;font-size:85%;line-height:1.45;background-color:#f6f8fa;border-radius:3px}
.markdown-body pre code{display:inline;max-width:auto;padding:0;margin:0;overflow:visible;line-height:inherit;word-wrap:normal;background-color:transparent;border:0}
.markdown-body pre code::after,.markdown-body pre code::before{content:normal}
.markdown-body .full-commit .btn-outline:not(:disabled):hover{color:#005cc5;border-color:#005cc5}
.markdown-body kbd{box-shadow:inset 0 -1px 0 #959da5;display:inline-block;padding:3px 5px;font:11px/10px SFMono-Regular,Consolas,"Liberation Mono",Menlo,Courier,monospace;color:#444d56;vertical-align:middle;background-color:#fcfcfc;border:1px solid #c6cbd1;border-bottom-color:#959da5;border-radius:3px;box-shadow:inset 0 -1px 0 #959da5}
.markdown-body :checked+.radio-label{position:relative;z-index:1;border-color:#0366d6}
.markdown-body .task-list-item{list-style-type:none}
.markdown-body .task-list-item+.task-list-item{margin-top:3px}
.markdown-body .task-list-item input{margin:0 .2em .25em -1.6em;vertical-align:middle}
.markdown-body::before{display:table;content:""}
.markdown-body::after{display:table;clear:both;content:""}
.markdown-body>:first-child{margin-top:0!important}
.markdown-body>:last-child{margin-bottom:0!important}
.Alert,.Error,.Note,.Success,.Warning,.Tips,.HTips{padding:11px;margin-bottom:24px;border-style:solid;border-width:1px;border-radius:4px}
.Alert p,.Error p,.Note p,.Success p,.Warning p,.Tips p,.HTips p{margin-top:0}
.Alert p:last-child,.Error p:last-child,.Note p:last-child,.Success p:last-child,.Warning p:last-child,.Tips p:last-child,.HTips p:last-child{margin-bottom:0}
.Alert{color:#246;background-color:#e2eef9;border-color:#bac6d3}
.Warning{color:#4c4a42;background-color:#fff9ea;border-color:#dfd8c2}
.Error{color:#911;background-color:#fcdede;border-color:#d2b2b2}
.Success{color:#22662c;background-color:#e2f9e5;border-color:#bad3be}
.Note{color:#2f363d;background-color:#f6f8fa;border-color:#d5d8da}
.Alert h1,.Alert h2,.Alert h3,.Alert h4,.Alert h5,.Alert h6{color:#246;margin-bottom:0}
.Warning h1,.Warning h2,.Warning h3,.Warning h4,.Warning h5,.Warning h6{color:#4c4a42;margin-bottom:0}
.Error h1,.Error h2,.Error h3,.Error h4,.Error h5,.Error h6{color:#911;margin-bottom:0}
.Success h1,.Success h2,.Success h3,.Success h4,.Success h5,.Success h6{color:#22662c;margin-bottom:0}
.Note h1,.Note h2,.Note h3,.Note h4,.Note h5,.Note h6{color:#2f363d;margin-bottom:0}
.Tips h1,.Tips h2,.Tips h3,.Tips h4,.Tips h5,.Tips h6{color:#2f363d;margin-bottom:0}
.HTips h1,.HTips h2,.HTips h3,.HTips h4,.HTips h5,.HTips h6{color:#2f363d;margin-bottom:0}
.Tips h1:first-child,.Tips h2:first-child,.Tips h3:first-child,.Tips h4:first-child,.Tips h5:first-child,.Tips h6:first-child,.Alert h1:first-child,.Alert h2:first-child,.Alert h3:first-child,.Alert h4:first-child,.Alert h5:first-child,.Alert h6:first-child,.Error h1:first-child,.Error h2:first-child,.Error h3:first-child,.Error h4:first-child,.Error h5:first-child,.Error h6:first-child,.Note h1:first-child,.Note h2:first-child,.Note h3:first-child,.Note h4:first-child,.Note h5:first-child,.Note h6:first-child,.Success h1:first-child,.Success h2:first-child,.Success h3:first-child,.Success h4:first-child,.Success h5:first-child,.Success h6:first-child,.Warning h1:first-child,.Warning h2:first-child,.Warning h3:first-child,.Warning h4:first-child,.Warning h5:first-child,.Warning h6:first-child{margin-top:0}
h1.title,p.subtitle{text-align:center}
h1.title.followed-by-subtitle{margin-bottom:0}
p.subtitle{font-size:1.5em;font-weight:600;line-height:1.25;margin-top:0;margin-bottom:16px;padding-bottom:.3em}
div.line-block{white-space:pre-line}
  </style>
  <style type="text/css">code{white-space: pre;}</style>
  <style type="text/css">
	pre > code.sourceCode { white-space: pre; position: relative; }
 pre > code.sourceCode > span { display: inline-block; line-height: 1.25; }
 pre > code.sourceCode > span:empty { height: 1.2em; }
 .sourceCode { overflow: visible; }
 code.sourceCode > span { color: inherit; text-decoration: inherit; }
 div.sourceCode { margin: 1em 0; }
 pre.sourceCode { margin: 0; }
 @media screen {
 div.sourceCode { overflow: auto; }
 }
 @media print {
 pre > code.sourceCode { white-space: pre-wrap; }
 pre > code.sourceCode > span { text-indent: -5em; padding-left: 5em; }
 }
 pre.numberSource code
   { counter-reset: source-line 0; }
 pre.numberSource code > span
   { position: relative; left: -4em; counter-increment: source-line; }
 pre.numberSource code > span > a:first-child::before
   { content: counter(source-line);
     position: relative; left: -1em; text-align: right; vertical-align: baseline;
     border: none; display: inline-block;
     -webkit-touch-callout: none; -webkit-user-select: none;
     -khtml-user-select: none; -moz-user-select: none;
     -ms-user-select: none; user-select: none;
     padding: 0 4px; width: 4em;
     background-color: #ffffff;
     color: #a0a0a0;
   }
 pre.numberSource { margin-left: 3em; border-left: 1px solid #a0a0a0;  padding-left: 4px; }
 div.sourceCode
   { color: #1f1c1b; background-color: #ffffff; }
 @media screen {
 pre > code.sourceCode > span > a:first-child::before { text-decoration: underline; }
 }
 code span { color: #1f1c1b; } /* Normal */
 code span.al { color: #bf0303; background-color: #f7e6e6; font-weight: bold; } /* Alert */
 code span.an { color: #ca60ca; } /* Annotation */
 code span.at { color: #0057ae; } /* Attribute */
 code span.bn { color: #b08000; } /* BaseN */
 code span.bu { color: #644a9b; font-weight: bold; } /* BuiltIn */
 code span.cf { color: #1f1c1b; font-weight: bold; } /* ControlFlow */
 code span.ch { color: #924c9d; } /* Char */
 code span.cn { color: #aa5500; } /* Constant */
 code span.co { color: #898887; } /* Comment */
 code span.cv { color: #0095ff; } /* CommentVar */
 code span.do { color: #607880; } /* Documentation */
 code span.dt { color: #0057ae; } /* DataType */
 code span.dv { color: #b08000; } /* DecVal */
 code span.er { color: #bf0303; text-decoration: underline; } /* Error */
 code span.ex { color: #0095ff; font-weight: bold; } /* Extension */
 code span.fl { color: #b08000; } /* Float */
 code span.fu { color: #644a9b; } /* Function */
 code span.im { color: #ff5500; } /* Import */
 code span.in { color: #b08000; } /* Information */
 code span.kw { color: #1f1c1b; font-weight: bold; } /* Keyword */
 code span.op { color: #1f1c1b; } /* Operator */
 code span.ot { color: #006e28; } /* Other */
 code span.pp { color: #006e28; } /* Preprocessor */
 code span.re { color: #0057ae; background-color: #e0e9f8; } /* RegionMarker */
 code span.sc { color: #3daee9; } /* SpecialChar */
 code span.ss { color: #ff5500; } /* SpecialString */
 code span.st { color: #bf0303; } /* String */
 code span.va { color: #0057ae; } /* Variable */
 code span.vs { color: #bf0303; } /* VerbatimString */
 code span.wa { color: #bf0303; } /* Warning */
  </style>
  <link rel="stylesheet" href="data:text/css,%3Aroot%20%7B%2D%2Dmain%2Ddarkblue%2Dcolor%3A%20rgb%283%2C35%2C75%29%3B%20%2D%2Dmain%2Dlightblue%2Dcolor%3A%20rgb%2860%2C180%2C230%29%3B%20%2D%2Dmain%2Dpink%2Dcolor%3A%20rgb%28230%2C0%2C126%29%3B%20%2D%2Dmain%2Dyellow%2Dcolor%3A%20rgb%28255%2C210%2C0%29%3B%20%2D%2Dsecondary%2Dgrey%2Dcolor%3A%20rgb%2870%2C70%2C80%29%3B%20%2D%2Dsecondary%2Dgrey%2Dcolor%2D25%3A%20rgb%28209%2C209%2C211%29%3B%20%2D%2Dsecondary%2Dgrey%2Dcolor%2D12%3A%20rgb%28233%2C233%2C234%29%3B%20%2D%2Dsecondary%2Dlightgreen%2Dcolor%3A%20rgb%2873%2C177%2C112%29%3B%20%2D%2Dsecondary%2Dpurple%2Dcolor%3A%20rgb%28140%2C0%2C120%29%3B%20%2D%2Dsecondary%2Ddarkgreen%2Dcolor%3A%20rgb%284%2C87%2C47%29%3B%20%2D%2Dsidenav%2Dfont%2Dsize%3A%2090%25%3B%7Dhtml%20%7Bfont%2Dfamily%3A%20%22Arial%22%2C%20sans%2Dserif%3B%7D%2A%20%7Bxbox%2Dsizing%3A%20border%2Dbox%3B%7D%2Est%5Fheader%20h1%2Etitle%2C%2Est%5Fheader%20p%2Esubtitle%20%7Btext%2Dalign%3A%20left%3B%7D%2Est%5Fheader%20h1%2Etitle%20%7Bborder%2Dcolor%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bmargin%2Dbottom%3A5px%3B%7D%2Est%5Fheader%20p%2Esubtitle%20%7Bcolor%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bfont%2Dsize%3A90%25%3B%7D%2Est%5Fheader%20h1%2Etitle%2Efollowed%2Dby%2Dsubtitle%20%7Bborder%2Dbottom%3A2px%20solid%3Bborder%2Dcolor%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bmargin%2Dbottom%3A5px%3B%7D%2Est%5Fheader%20p%2Erevision%20%7Bdisplay%3A%20inline%2Dblock%3Bwidth%3A70%25%3B%7D%2Est%5Fheader%20div%2Eauthor%20%7Bfont%2Dstyle%3A%20italic%3B%7D%2Est%5Fheader%20div%2Esummary%20%7Bborder%2Dtop%3A%20solid%201px%20%23C0C0C0%3Bbackground%3A%20%23ECECEC%3Bpadding%3A%205px%3B%7D%2Est%5Ffooter%20%7Bfont%2Dsize%3A80%25%3B%7D%2Est%5Ffooter%20img%20%7Bfloat%3A%20right%3B%7D%2Est%5Ffooter%20%2Est%5Fnotice%20%7Bwidth%3A80%25%3B%7D%2Emarkdown%2Dbody%20%23header%2Dsection%2Dnumber%20%7Bfont%2Dsize%3A120%25%3B%7D%2Emarkdown%2Dbody%20h1%20%7Bborder%2Dbottom%3A1px%20solid%3Bborder%2Dcolor%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bpadding%2Dbottom%3A%202px%3Bpadding%2Dtop%3A%2010px%3B%7D%2Emarkdown%2Dbody%20h2%20%7Bpadding%2Dbottom%3A%205px%3Bpadding%2Dtop%3A%2010px%3B%7D%2Emarkdown%2Dbody%20h2%20code%20%7Bbackground%2Dcolor%3A%20rgb%28255%2C%20255%2C%20255%29%3B%7D%23func%2EsourceCode%20%7Bborder%2Dleft%2Dstyle%3A%20solid%3Bborder%2Dcolor%3A%20rgb%280%2C%2032%2C%2082%29%3Bborder%2Dcolor%3A%20rgb%28255%2C%20244%2C%20191%29%3Bborder%2Dwidth%3A%208px%3Bpadding%3A0px%3B%7Dpre%20%3E%20code%20%7Bborder%3A%20solid%201px%20blue%3Bfont%2Dsize%3A60%25%3B%7DcodeXX%20%7Bborder%3A%20solid%201px%20blue%3Bfont%2Dsize%3A60%25%3B%7D%23func%2EsourceXXCode%3A%3Abefore%20%7Bcontent%3A%20%22Synopsis%22%3Bpadding%2Dleft%3A10px%3Bfont%2Dweight%3A%20bold%3B%7Dfigure%20%7Bpadding%3A0px%3Bmargin%2Dleft%3A5px%3Bmargin%2Dright%3A5px%3Bmargin%2Dleft%3A%20auto%3Bmargin%2Dright%3A%20auto%3B%7Dimg%5Bdata%2Dproperty%3D%22center%22%5D%20%7Bdisplay%3A%20block%3Bmargin%2Dtop%3A%2010px%3Bmargin%2Dleft%3A%20auto%3Bmargin%2Dright%3A%20auto%3Bpadding%3A%2010px%3B%7Dfigcaption%20%7Btext%2Dalign%3Aleft%3B%20%20border%2Dtop%3A%201px%20dotted%20%23888%3Bpadding%2Dbottom%3A%2020px%3Bmargin%2Dtop%3A%2010px%3B%7Dh1%20code%2C%20h2%20code%20%7Bfont%2Dsize%3A120%25%3B%7D%09%2Emarkdown%2Dbody%20table%20%7Bwidth%3A%20100%25%3Bmargin%2Dleft%3Aauto%3Bmargin%2Dright%3Aauto%3B%7D%2Emarkdown%2Dbody%20img%20%7Bborder%2Dradius%3A%204px%3Bpadding%3A%205px%3Bdisplay%3A%20block%3Bmargin%2Dleft%3A%20auto%3Bmargin%2Dright%3A%20auto%3Bwidth%3A%20auto%3B%7D%2Emarkdown%2Dbody%20%2Est%5Fheader%20img%2C%20%2Emarkdown%2Dbody%20%7Bborder%3A%20none%3Bborder%2Dradius%3A%20none%3Bpadding%3A%205px%3Bdisplay%3A%20block%3Bmargin%2Dleft%3A%20auto%3Bmargin%2Dright%3A%20auto%3Bwidth%3A%20auto%3Bbox%2Dshadow%3A%20none%3B%7D%2Emarkdown%2Dbody%20%7Bmargin%3A%2010px%3Bpadding%3A%2010px%3Bwidth%3A%20auto%3Bfont%2Dfamily%3A%20%22Arial%22%2C%20sans%2Dserif%3Bcolor%3A%20%2303234B%3Bcolor%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%7D%2Emarkdown%2Dbody%20h1%2C%20%2Emarkdown%2Dbody%20h2%2C%20%2Emarkdown%2Dbody%20h3%20%7B%20%20%20color%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%7D%2Emarkdown%2Dbody%3Ahover%20%7B%7D%2Emarkdown%2Dbody%20%2Econtents%20%7B%7D%2Emarkdown%2Dbody%20%2Etoc%2Dtitle%20%7B%7D%2Emarkdown%2Dbody%20%2Econtents%20li%20%7Blist%2Dstyle%2Dtype%3A%20none%3B%7D%2Emarkdown%2Dbody%20%2Econtents%20ul%20%7Bpadding%2Dleft%3A%2010px%3B%7D%2Emarkdown%2Dbody%20%2Econtents%20a%20%7Bcolor%3A%20%233CB4E6%3B%20%7D%2Emarkdown%2Dbody%20table%20%2Eheader%20%7Bbackground%2Dcolor%3A%20var%28%2D%2Dsecondary%2Dgrey%2Dcolor%2D12%29%3Bborder%2Dbottom%3A1px%20solid%3Bborder%2Dtop%3A1px%20solid%3Bfont%2Dsize%3A%2090%25%3B%7D%2Emarkdown%2Dbody%20table%20th%20%7Bfont%2Dweight%3A%20bolder%3B%20%7D%2Emarkdown%2Dbody%20table%20td%20%7Bfont%2Dsize%3A%2090%25%3B%7D%2Emarkdown%2Dbody%20code%7Bpadding%3A%200%3Bmargin%3A0%3Bfont%2Dsize%3A95%25%3Bbackground%2Dcolor%3Argba%2827%2C31%2C35%2C%2E05%29%3Bborder%2Dradius%3A1px%3B%7D%2Et01%20%7Bwidth%3A%20100%25%3Bborder%3A%20None%3Btext%2Dalign%3A%20left%3B%7D%2ETips%20%7Bpadding%3A11px%3Bmargin%2Dbottom%3A24px%3Bborder%2Dstyle%3Asolid%3Bborder%2Dwidth%3A1px%3Bborder%2Dradius%3A1px%7D%2ETips%20%7Bcolor%3A%232f363d%3B%20background%2Dcolor%3A%20%23f6f8fa%3Bborder%2Dcolor%3A%23d5d8da%3Bborder%2Dtop%3A1px%20solid%3Bborder%2Dbottom%3A1px%20solid%3B%7D%2EHTips%20%7Bpadding%3A11px%3Bmargin%2Dbottom%3A24px%3Bborder%2Dstyle%3Asolid%3Bborder%2Dwidth%3A1px%3Bborder%2Dradius%3A1px%7D%2EHTips%20%7Bcolor%3A%232f363d%3B%20background%2Dcolor%3A%23fff9ea%3Bborder%2Dcolor%3A%23d5d8da%3Bborder%2Dtop%3A1px%20solid%3Bborder%2Dbottom%3A1px%20solid%3B%7D%2EHTips%20h1%2C%2EHTips%20h2%2C%2EHTips%20h3%2C%2EHTips%20h4%2C%2EHTips%20h5%2C%2EHTips%20h6%20%7Bcolor%3A%232f363d%3Bmargin%2Dbottom%3A0%7D%2Esidenav%20%7Bfont%2Dfamily%3A%20%22Arial%22%2C%20sans%2Dserif%3B%20%20color%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bheight%3A%20100%25%3Bposition%3A%20fixed%3Bz%2Dindex%3A%201%3Btop%3A%200%3Bleft%3A%200%3Bmargin%2Dright%3A%2010px%3Bmargin%2Dleft%3A%2010px%3B%20overflow%2Dx%3A%20hidden%3B%7D%2Esidenav%20hr%2Enew1%20%7Bborder%2Dwidth%3A%20thin%3Bborder%2Dcolor%3A%20var%28%2D%2Dmain%2Dlightblue%2Dcolor%29%3Bmargin%2Dright%3A%2010px%3Bmargin%2Dtop%3A%20%2D10px%3B%7D%2Esidenav%20%23sidenav%5Fheader%20%7Bmargin%2Dtop%3A%2010px%3Bborder%3A%201px%3Bcolor%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bborder%2Dcolor%3A%20var%28%2D%2Dmain%2Dlightblue%2Dcolor%29%3B%7D%2Esidenav%20%23sidenav%5Fheader%20img%20%7Bfloat%3A%20left%3B%7D%2Esidenav%20%23sidenav%5Fheader%20a%20%7Bmargin%2Dleft%3A%200px%3Bmargin%2Dright%3A%200px%3Bpadding%2Dleft%3A%200px%3B%7D%2Esidenav%20%23sidenav%5Fheader%20a%3Ahover%20%7Bbackground%2Dsize%3A%20auto%3Bcolor%3A%20%23FFD200%3B%20%7D%2Esidenav%20%23sidenav%5Fheader%20a%3Aactive%20%7B%20%20%7D%2Esidenav%20%3E%20ul%20%7Bbackground%2Dcolor%3A%20rgba%2857%2C%20169%2C%20220%2C%200%2E05%29%3B%20color%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bborder%2Dradius%3A%2010px%3Bpadding%2Dbottom%3A%2010px%3Bpadding%2Dtop%3A%2010px%3Bpadding%2Dright%3A%2010px%3Bmargin%2Dright%3A%2010px%3B%7D%2Esidenav%20a%20%7Bpadding%3A%202px%202px%3Btext%2Ddecoration%3A%20none%3Bfont%2Dsize%3A%20var%28%2D%2Dsidenav%2Dfont%2Dsize%29%3Bdisplay%3Atable%3B%7D%2Esidenav%20%3E%20ul%20%3E%20li%2C%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%7B%20padding%2Dright%3A%205px%3Bpadding%2Dleft%3A%205px%3B%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20a%20%7B%20color%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bfont%2Dweight%3A%20lighter%3B%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%3E%20a%20%7B%20color%3A%20var%28%2D%2Dmain%2Ddarkblue%2Dcolor%29%3Bfont%2Dsize%3A%2080%25%3Bpadding%2Dleft%3A%2010px%3Btext%2Dalign%2Dlast%3A%20left%3B%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%3E%20a%20%7B%20display%3A%20None%3B%7D%2Esidenav%20li%20%7Blist%2Dstyle%2Dtype%3A%20none%3B%7D%2Esidenav%20ul%20%7Bpadding%2Dleft%3A%200px%3B%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20a%3Ahover%2C%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%3E%20a%3Ahover%20%7Bbackground%2Dcolor%3A%20var%28%2D%2Dsecondary%2Dgrey%2Dcolor%2D12%29%3Bbackground%2Dclip%3A%20border%2Dbox%3Bmargin%2Dleft%3A%20%2D10px%3Bpadding%2Dleft%3A%2010px%3B%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20a%3Ahover%20%7Bpadding%2Dright%3A%2015px%3Bwidth%3A%20230px%3B%09%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%3E%20a%3Ahover%20%7Bpadding%2Dright%3A%2010px%3Bwidth%3A%20230px%3B%09%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20a%3Aactive%20%7B%20color%3A%20%23FFD200%3B%20%7D%2Esidenav%20%3E%20ul%20%3E%20li%20%3E%20ul%20%3E%20li%20%3E%20a%3Aactive%20%7B%20color%3A%20%23FFD200%3B%20%7D%2Esidenav%20code%20%7B%7D%2Esidenav%20%7Bwidth%3A%20280px%3B%7D%23sidenav%20%7Bmargin%2Dleft%3A%20300px%3Bdisplay%3Ablock%3B%7D%2Emarkdown%2Dbody%20%2Eprint%2Dcontents%20%7Bvisibility%3Ahidden%3B%7D%2Emarkdown%2Dbody%20%2Eprint%2Dtoc%2Dtitle%20%7Bvisibility%3Ahidden%3B%7D%2Emarkdown%2Dbody%20%7Bmax%2Dwidth%3A%20980px%3Bmin%2Dwidth%3A%20200px%3Bpadding%3A%2040px%3Bborder%2Dstyle%3A%20solid%3Bborder%2Dstyle%3A%20outset%3Bborder%2Dcolor%3A%20rgba%28104%2C%20167%2C%20238%2C%200%2E089%29%3Bborder%2Dradius%3A%205px%3B%7D%40media%20screen%20and%20%28max%2Dheight%3A%20450px%29%20%7B%2Esidenav%20%7Bpadding%2Dtop%3A%2015px%3B%7D%2Esidenav%20a%20%7Bfont%2Dsize%3A%2018px%3B%7D%23sidenav%20%7Bmargin%2Dleft%3A%2010px%3B%20%7D%2Esidenav%20%7Bvisibility%3Ahidden%3B%7D%2Emarkdown%2Dbody%20%7Bmargin%3A%2010px%3Bpadding%3A%2040px%3Bwidth%3A%20auto%3Bborder%3A%200px%3B%7D%7D%40media%20screen%20and%20%28max%2Dwidth%3A%201024px%29%20%7B%2Esidenav%20%7Bvisibility%3Ahidden%3B%7D%2Emarkdown%2Dbody%20%7Bmargin%3A%2010px%3Bpadding%3A%2040px%3Bwidth%3A%20auto%3Bborder%3A%200px%3B%7D%23sidenav%20%7Bmargin%2Dleft%3A%2010px%3B%7D%7D%40media%20print%20%7B%2Esidenav%20%7Bvisibility%3Ahidden%3B%7D%23sidenav%20%7Bmargin%2Dleft%3A%2010px%3B%7D%2Emarkdown%2Dbody%20%7Bmargin%3A%2010px%3Bpadding%3A%2010px%3Bwidth%3Aauto%3Bborder%3A%200px%3B%7D%40page%20%7Bsize%3A%20A4%3B%20%20margin%3A2cm%3Bpadding%3A2cm%3Bmargin%2Dtop%3A%201cm%3Bpadding%2Dbottom%3A%201cm%3B%7D%2A%20%7Bxbox%2Dsizing%3A%20border%2Dbox%3Bfont%2Dsize%3A90%25%3B%7Da%20%7Bfont%2Dsize%3A%20100%25%3Bcolor%3A%20yellow%3B%7D%2Emarkdown%2Dbody%20article%20%7Bxbox%2Dsizing%3A%20border%2Dbox%3Bfont%2Dsize%3A100%25%3B%7D%2Emarkdown%2Dbody%20p%20%7Bwindows%3A%202%3Borphans%3A%202%3B%7D%2Epagebreakerafter%20%7Bpage%2Dbreak%2Dafter%3A%20always%3Bpadding%2Dtop%3A10mm%3B%7D%2Epagebreakbefore%20%7Bpage%2Dbreak%2Dbefore%3A%20always%3B%7Dh1%2C%20h2%2C%20h3%2C%20h4%20%7Bpage%2Dbreak%2Dafter%3A%20avoid%3B%7Ddiv%2C%20code%2C%20blockquote%2C%20li%2C%20span%2C%20table%2C%20figure%20%7Bpage%2Dbreak%2Dinside%3A%20avoid%3B%7D%7D">
  <!--[if lt IE 9]>
    <script src="//cdnjs.cloudflare.com/ajax/libs/html5shiv/3.7.3/html5shiv-printshiv.min.js"></script>
  <![endif]-->





<link rel="shortcut icon" href="">

</head>



<body>

		<div class="sidenav">
		<div id="sidenav_header">
							<img src="" title="STM32CubeMX.AI logo" align="left" height="70" />
										<br />7.0.0<br />
										<a href="#doc_title"> Evaluation report and metrics </a>
					</div>
		<div id="sidenav_header_button">
			 
							<ul>
					<li><p><a id="index" href="index.html">[ Index ]</a></p></li>
				</ul>
						<hr class="new1">
		</div>	

		<ul>
  <li><a href="#purpose">Purpose</a>
  <ul>
  <li><a href="#evaluated-metrics">Evaluated metrics</a></li>
  <li><a href="#computation-of-the-metrics">Computation of the metrics</a></li>
  <li><a href="#and-for-the-quantized-models">And for the quantized models?</a></li>
  <li><a href="#noexec-model-option">“–no–exec-model” option</a></li>
  <li><a href="#ref_data_recom">Specific attention on the provided data</a></li>
  <li><a href="#ref_data_random">Random data generation</a></li>
  <li><a href="#ref_regressor_model">Classifier and regressor models</a></li>
  <li><a href="#ref_valio_arg">Input validation files</a></li>
  <li><a href="#ref_post_proc_support">Output validation files for post-processing</a></li>
  </ul></li>
  <li><a href="#metrics">Metrics</a>
  <ul>
  <li><a href="#ref_complexity">Computational complexity: MACC and cycles/MACC</a></li>
  <li><a href="#ref_memory_occupancy">Memory-related metrics</a></li>
  <li><a href="#ref_acc">Classification accuracy (acc)</a></li>
  <li><a href="#ref_rmse">Root Mean Square Error (rmse)</a></li>
  <li><a href="#ref_mae">Mean Absolute Error (mae)</a></li>
  <li><a href="#ref_l2r">L2 relative error (l2r)</a></li>
  <li><a href="#ref_mean">Arithmetic mean of the error (mean)</a></li>
  <li><a href="#ref_std">Standard deviation of the error (std)</a></li>
  <li><a href="#ref_cm">Confusion matrix (CM)</a></li>
  </ul></li>
  <li><a href="#metric_interpretation">Interpretation of the results</a>
  <ul>
  <li><a href="#floating-point-model">Floating point model</a></li>
  <li><a href="#quantized-model">Quantized model</a></li>
  <li><a href="#quantized-keras-model">Quantized keras model</a></li>
  <li><a href="#model-with-multiple-io">Model with multiple IO</a></li>
  </ul></li>
  <li><a href="#ref_script_ex">Post-processing example</a></li>
  <li><a href="#references">References</a></li>
  </ul>
	</div>
	<article id="sidenav" class="markdown-body">
		



<header>
<section class="st_header" id="doc_title">

<div class="himage">
	<img src="" title="STM32CubeMX.AI" align="right" height="70" />
	<img src="" title="STM32" align="right" height="90" />
</div>

<h1 class="title followed-by-subtitle">Evaluation report and metrics</h1>

	<p class="subtitle">X-CUBE-AI Expansion Package</p>

	<div class="revision">r2.1</div>

	<div class="ai_platform">
		AI PLATFORM r7.0.0
					(Embedded Inference Client API 1.1.0)
			</div>
			Command Line Interface r1.5.1
	




</section>
</header>
 




<section id="purpose" class="level1">
<h1>Purpose</h1>
<p>This article describes the different metrics (and associated computing flow) which are used to evaluate the performance of the generated C-files (or C-model) mainly through the <a href="command_line_interface.html#validate-command">validate</a> command. Proposed metrics should be considered as the <em>generic indicators</em> which allows to compare numerically the predictions of the c-model against the predictions of the original model. Only the simple scalar values are computed, no specific threshold or pre/post-process are used. For <a href="#ref_script_ex">post-process evaluation</a>, all injected and predicted data (including the data from the original model) are saved or the end-user can also <a href="how_to_run_a_model_locally.html">executed locally the c-model</a> for efficient and advanced or customized validation flow.</p>
<div class="Alert">
<p><strong>Warning</strong> — Be aware, that the underlying validation engine has not been designed and optimized, in term of execution time and host resource usage, to valid a pre-trained model as during a training/test phase. A representative and small sub-set of the whole training data set is expected to test the exactness of the generated C-model running on the desktop/X86 or STM32 run-time.</p>
</div>
<section id="evaluated-metrics" class="level2">
<h2>Evaluated metrics</h2>
<table>
<colgroup>
<col style="width: 7%" />
<col style="width: 9%" />
<col style="width: 40%" />
<col style="width: 42%" />
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">metric</th>
<th style="text-align: left;">category</th>
<th style="text-align: left;">description</th>
<th style="text-align: left;">applicable to</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">MACC</td>
<td style="text-align: left;">complexity</td>
<td style="text-align: left;"><a href="#ref_complexity">computational complexity</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="even">
<td style="text-align: left;">ROM/RAM</td>
<td style="text-align: left;">memory</td>
<td style="text-align: left;"><a href="#ref_memory_occupancy">memory-related metrics</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;">ACC</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_acc">accuracy (Classification accuracy)</a></td>
<td style="text-align: left;">only classifier models (float and integer format)</td>
</tr>
<tr class="even">
<td style="text-align: left;">RMSE</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_rmse">Root Mean Square Error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;">MAE</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_mae">Mean Absolute Error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="even">
<td style="text-align: left;">L2r</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_l2r">L2 relative Error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;">MEAN</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_mean">Arithmetic mean of the error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="even">
<td style="text-align: left;">STD</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_std">Standard deviation of the error</a></td>
<td style="text-align: left;">all models</td>
</tr>
<tr class="odd">
<td style="text-align: left;">CM</td>
<td style="text-align: left;">perf</td>
<td style="text-align: left;"><a href="#ref_cm">Confusion Matrix</a></td>
<td style="text-align: left;">only classifier models (float and integer format)</td>
</tr>
</tbody>
</table>
</section>
<section id="computation-of-the-metrics" class="level2">
<h2>Computation of the metrics</h2>
<p>The MACC and ROM/RAM metrics are computed during the import of the model. The other metrics are evaluated during the process of validation (i.e. <a href="command_line_interface.html#validate-command">validate</a> command). By default, no user data are requested (default mode), the models are feed with the <a href="#ref_data_random">random data</a>. However, to be more precise, the user has the possibility to give a representative pre-processed data set (with or without the references). Raw input and output data are always saved in a <a href="#ref_post_proc_support">dedicated file</a> which could be used by a post-process user script.</p>
<div id="fig:id_m_compute_def" class="fignos">
<figure>
<img src="" property="center" style="width:90.0%" alt="Figure 1: Computation of the metrics (default mode)" /><figcaption aria-hidden="true"><span>Figure 1:</span> Computation of the metrics (default mode)</figcaption>
</figure>
</div>
<div id="fig:id_m_compute" class="fignos">
<figure>
<img src="" property="center" style="width:90.0%" alt="Figure 2: Computation of the metrics (with user data)" /><figcaption aria-hidden="true"><span>Figure 2:</span> Computation of the metrics (with user data)</figcaption>
</figure>
</div>
<ul>
<li><strong>[ I ]</strong> designates the list of the pre-processed samples (or inputs) which are used to feed the original model and the C-model. It can be provided by the user (see <a href="#ref_valio_arg">“Inputs validation files”</a> and <a href="#ref_data_recom">“Specific attention on the provided data”</a> sections) or randomly generated (see <a href="#ref_data_random">“Random data generation”</a> section). They are used as-is without pre-processing with a potential exception for the quantized models.</li>
<li><strong>[ P ]</strong> designates the list of the predicted samples inferred by the C-model.</li>
<li><strong>[ R’ ]</strong> designates the list of the predicted samples inferred by the original model.</li>
<li>{optional} <strong>[ R ]</strong> designates, the list of the predicted output samples provided by the user which will be used as ground truth or reference values.</li>
</ul>
<p>At the end of the process, metrics are summarized in a simple table.</p>
<pre><code>Evaluation report (summary)
----------------------------------------------------------------------------------------------------------
Mode                 acc      rmse      mae       l2r       tensor
----------------------------------------------------------------------------------------------------------
x86 C-model #1       92.68%   0.053623  0.005785  0.340042  dense_4_nl [ai_float, [(1, 1, 36)], m_id=[10]]
original model #1    92.68%   0.053623  0.005785  0.340042  dense_4_nl [ai_float, [(1, 1, 36)], m_id=[10]]
X-cross #1           100.00%  0.000000  0.000000  0.000000  dense_4_nl [ai_float, [(1, 1, 36)], m_id=[10]]
----------------------------------------------------------------------------------------------------------</code></pre>
<ul>
<li><em>X-cross #1</em> indicates the metrics which are evaluated with the <strong>[ P ]</strong> and <strong>[ R’ ]</strong> data for the first output #1 (there is one line by output). In this case, the predicted values <strong>[ R’ ]</strong> are considered as the <strong>references</strong>.</li>
<li><em>x86 [or stm32] C-model</em> (respectively <em>original model</em>) designates the case where <strong>[ R ]</strong> data are also provided by the user allowing to compute the metrics against theses references or ground truth values. If <strong>[ R ]</strong> is not provided only <em>X-cross #1</em> metrics are computed.</li>
</ul>
<p>When the generated c-model is executed on the STM32 device through the AI Validation project (refer to <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">[UM]</a>), the same built-in validation flow is applied.</p>
<div id="fig:id_val_stm32" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt="Figure 3: Computation of the metric (STM32 case)" /><figcaption aria-hidden="true"><span>Figure 3:</span> Computation of the metric (STM32 case)</figcaption>
</figure>
</div>
</section>
<section id="and-for-the-quantized-models" class="level2">
<h2>And for the quantized models?</h2>
<p>No specific metrics are defined for the quantized models. The same metrics are processed for a quantized model or not, with or without input/output in integer. Be aware that the metrics are <strong>always</strong> computed with the <code>float32</code> data types. If the data type is <code>int8</code> (or <code>uint8</code>), data are de-quantized before. The same scale/zero-point values from the original model are used for all the data: <strong>[ P ]</strong>, <strong>[ R’ ]</strong> and <strong>[ R ]</strong>. Symmetrically, if <code>float32</code> data type are provided, data are quantized before to feed the model.</p>
<div id="fig:id_val_quant" class="fignos">
<figure>
<img src="" property="center" style="width:90.0%" alt="Figure 4: Computation of the metrics (quantized models)" /><figcaption aria-hidden="true"><span>Figure 4:</span> Computation of the metrics (quantized models)</figcaption>
</figure>
</div>
<div class="HTips">
<p><strong>Note</strong> — If <code>uint8</code> data type are provided, and the requested quantized model expects <code>int8</code> or <code>float32</code> type, an exception will be raised. Data importer is not able to convert the data to the expected format.</p>
</div>
</section>
<section id="noexec-model-option" class="level2">
<h2>“–no–exec-model” option</h2>
<p>The <code>&#39;--no-exec-model&#39;</code> option allows to perform the built-in validation flow w/o the execution of the original model. This can be considered as a workaround when sometimes a particular model can be imported and generated but the associated inference engine embedded in the pack is not compatible or older. The other interest is to improve the execution time when only the predicted data by the c-model are requested.</p>
<div id="fig:id_val_no_exec" class="fignos">
<figure>
<img src="" property="center" style="width:90.0%" alt="Figure 5: –no-exec-model option" /><figcaption aria-hidden="true"><span>Figure 5:</span> –no-exec-model option</figcaption>
</figure>
</div>
</section>
<section id="ref_data_recom" class="level2">
<h2>Specific attention on the provided data</h2>
<p>To have an accurate built-in validation process or more significant metrics, it is important to feed the original model and the generated c-model with closest data from the validating data set used to test the original model. If the raw data set have been pre-processed, the user should build a representative data set with a sub-set of these pre-processed data.</p>
<div id="fig:id_input_creation" class="fignos">
<figure>
<img src="" property="center" style="width:75.0%" alt="Figure 6: Representative data set creation" /><figcaption aria-hidden="true"><span>Figure 6:</span> Representative data set creation</figcaption>
</figure>
</div>
<p>This recommendation is also applicable for the output data. For the computation of the metrics are based on the element-wise operations between the provided references and the predicted values. For a classifier (one or multiple classes) for example, one-hot encoding data can be provided. Integer Encoding is not supported.</p>
<p>The following figure illustrates a typical example, where a particular attention is requested. The STM32Cube function pack for computer vision (<a href="https://www.st.com/en/embedded-software/fp-ai-vision1.html">https://www.st.com/en/embedded-software/fp-ai-vision1.html</a>) provides an advanced debug mode which allows to inject or to dump the pre-processed images. If the user wants to valid its pre-trained model with the X-CUBE-AI validation engine and to compare with the deployed model, he must ensure that the format and the applied pre-processing are similar to have the more accurate validation metrics. In this use-case, to take account the pre-processing done by the pack, it is recommended to create a set of representative pre-processed images to test different models off-line or to refine an existing pre-trained model.</p>
<div id="fig:id_fp_vision" class="fignos">
<figure>
<img src="" property="center" style="width:95.0%" alt="Figure 7: Validation against the FP-Vision" /><figcaption aria-hidden="true"><span>Figure 7:</span> Validation against the FP-Vision</figcaption>
</figure>
</div>
</section>
<section id="ref_data_random" class="level2">
<h2>Random data generation</h2>
<p>When no user data are provided, random data are generated. By default, the value are uniformly distributed with a <em>fixed seed</em> in the range <code>[0.0, 1.0[</code> to have a reproducible test. The user has the possibility to change the min and max values with the <code>--range</code> option. During the validation process, min/max/mean and std values for the different input and outputs are reported. The <code>--seed</code> option can be also to change the initial seed.</p>
<p>To generate a batch of 20 samples by inputs with the data uniformly distributed between -10 and 5.</p>
<div class="sourceCode" id="cb2"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb2-1"><a href="#cb2-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>model_file<span class="op">&gt;</span> <span class="op">--</span>range <span class="op">-</span>10 5 <span class="op">-</span>b 20</span>
<span id="cb2-2"><a href="#cb2-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb2-3"><a href="#cb2-3" aria-hidden="true" tabindex="-1"></a>Setting validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb2-4"><a href="#cb2-4" aria-hidden="true" tabindex="-1"></a> generating random <span class="kw">data</span><span class="op">,</span> size<span class="op">=</span>20<span class="op">,</span> seed<span class="op">=</span>42<span class="op">,</span> range<span class="op">=(-</span>10<span class="op">.</span><span class="fu">0</span><span class="op">,</span> 5<span class="op">.</span><span class="fu">0</span><span class="op">)</span></span>
<span id="cb2-5"><a href="#cb2-5" aria-hidden="true" tabindex="-1"></a> I<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>20<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 99<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[-</span>9<span class="op">.</span><span class="fu">952</span><span class="op">,</span> 4<span class="op">.</span><span class="fu">996</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[-</span>2<span class="op">.</span><span class="fu">513</span><span class="op">,</span> 4<span class="op">.</span><span class="fu">383</span><span class="op">],</span> input_0</span>
<span id="cb2-6"><a href="#cb2-6" aria-hidden="true" tabindex="-1"></a> No output<span class="op">/</span>reference samples are provided</span>
<span id="cb2-7"><a href="#cb2-7" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
<p>If the model has the inputs in integer (<code>&#39;int8&#39;</code> or <code>&#39;uint8&#39;</code>), the min and max value are automatically inferred with the associated scale and zero-point values to be sure to have an uniform distribution between the min/max values of the input data type (i.e. <code>[-128, 127]</code> for <code>int8</code> type).</p>
<div class="sourceCode" id="cb3"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb3-1"><a href="#cb3-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>quant_model_file<span class="op">&gt;</span></span>
<span id="cb3-2"><a href="#cb3-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb3-3"><a href="#cb3-3" aria-hidden="true" tabindex="-1"></a>Setting validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb3-4"><a href="#cb3-4" aria-hidden="true" tabindex="-1"></a> generating random <span class="kw">data</span><span class="op">,</span> size<span class="op">=</span>10<span class="op">,</span> seed<span class="op">=</span>42<span class="op">,</span> range<span class="op">=</span>default</span>
<span id="cb3-5"><a href="#cb3-5" aria-hidden="true" tabindex="-1"></a> I<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>10<span class="op">,</span> 49<span class="op">,</span> 40<span class="op">,</span> 1<span class="op">)/</span>uint8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">,</span> 255<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>127<span class="op">.</span><span class="fu">323</span><span class="op">,</span> 73<span class="op">.</span><span class="fu">590</span><span class="op">],</span> scale<span class="op">=</span>0<span class="op">.</span><span class="fu">10196070</span> zp<span class="op">=</span>0<span class="op">,</span> Reshape_1</span>
<span id="cb3-6"><a href="#cb3-6" aria-hidden="true" tabindex="-1"></a> No output<span class="op">/</span>reference samples are provided</span>
<span id="cb3-7"><a href="#cb3-7" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
</section>
<section id="ref_regressor_model" class="level2">
<h2>Classifier and regressor models</h2>
<p>No specific metrics are defined for the regressor models. <a href="#ref_acc">ACC</a> and <a href="#ref_cm">CM</a> metrics are only evaluated if the predicted outputs <strong>[ R’ ]</strong> (or <strong>[ R ]</strong>) may represent probabilities within a given tolerance. Nevertheless, the <code>&#39;--classifier&#39;</code> option can be used to force the computation of the <code>&#39;ACC</code> and <code>CM&#39;</code> metrics.</p>
<div class="sourceCode" id="cb4"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb4-1"><a href="#cb4-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">-</span>m <span class="op">&lt;</span>regressor_model_file<span class="op">&gt;</span></span>
<span id="cb4-2"><a href="#cb4-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb4-3"><a href="#cb4-3" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb4-4"><a href="#cb4-4" aria-hidden="true" tabindex="-1"></a><span class="op">-------------------------------------------------------------------------------------------------------</span></span>
<span id="cb4-5"><a href="#cb4-5" aria-hidden="true" tabindex="-1"></a>Mode         acc    rmse          mae           l2r           tensor</span>
<span id="cb4-6"><a href="#cb4-6" aria-hidden="true" tabindex="-1"></a><span class="op">-------------------------------------------------------------------------------------------------------</span></span>
<span id="cb4-7"><a href="#cb4-7" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1   n.a.   0.000000065   0.000000048   0.000000127   dense_2, ai_float, [(1, 1, 1)], m_id=[2]</span></span>
<span id="cb4-8"><a href="#cb4-8" aria-hidden="true" tabindex="-1"></a><span class="op">-------------------------------------------------------------------------------------------------------</span></span></code></pre></div>
</section>
<section id="ref_valio_arg" class="level2">
<h2>Input validation files</h2>
<p>The user can provide the inputs and associated ground truth or reference values in a simple file (npz file) or separated files (npy or csv files). During the import of the data, each array is reshaped according the shape of the input (respectively output) of the c-model.</p>
<table>
<colgroup>
<col style="width: 16%" />
<col style="width: 83%" />
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">file format</th>
<th style="text-align: left;">description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">csv</td>
<td style="text-align: left;">Text files with a flattened version of the input (or output) tensors. One sample by line is expected. Comma <code>&#39;,&#39;</code> separator is used to separate the values.</td>
</tr>
<tr class="even">
<td style="text-align: left;">npy</td>
<td style="text-align: left;">Simple binary numpy file with a single array. (batch-size, -1) shape.</td>
</tr>
<tr class="odd">
<td style="text-align: left;">pb</td>
<td style="text-align: left;">Simple binary TensorProto file with a single array. (batch-size, -1) shape. Can be create with the <code>tf.make_tensor_proto</code> helper function.</td>
</tr>
<tr class="even">
<td style="text-align: left;">npz</td>
<td style="text-align: left;">Binary numpy file with several arrays.</td>
</tr>
</tbody>
</table>
<p>For the <strong>npz</strong> file, the following pairs of keys (dict entries) are supported (see the <a href="faq_validation.html#ref_npz_images">snippet code</a> in FAQ article to generate a <code>npz</code> file from a image data set):</p>
<ul>
<li><code>x_test</code> and <code>y_test</code> (simple IO)</li>
<li><code>inputs</code> and <code>outputs</code> (simple IO)</li>
<li><code>in_0</code> and <code>out_0</code> (simple IO)</li>
<li><code>m_inputs</code> and <code>m_outputs</code> (simple IO)</li>
<li><code>m_inputs_&lt;idx&gt;</code> and <code>m_outputs_&lt;idx&gt;</code> (multiple IO, <code>idx</code> starting with 1)</li>
<li><code>c_inputs_&lt;idx&gt;</code> and <code>c_outputs_&lt;idx&gt;</code> (multiple IO, if no <code>m_inputs_&lt;idx&gt;</code> keys are defined)</li>
<li>else <code>xxx</code> keys are considered as the multiple or simple inputs, consequently no outputs will be considered</li>
</ul>
<p>For the <strong>csv</strong> files, a particular tag (<code>&#39;dtype=uint8&#39;</code> or <code>&#39;dtype=int8&#39;</code>) must be defined inside the five first comment lines to indicate the type of the data, else float32 data type will be used.</p>
<div class="sourceCode" id="cb5"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb5-1"><a href="#cb5-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Example of csv file (32-b float number)</span></span>
<span id="cb5-2"><a href="#cb5-2" aria-hidden="true" tabindex="-1"></a><span class="co"># comment line</span></span>
<span id="cb5-3"><a href="#cb5-3" aria-hidden="true" tabindex="-1"></a><span class="op">-</span>1<span class="op">.</span><span class="fu">076007485389709473e</span><span class="op">+</span>00<span class="op">,</span>6<span class="op">.</span><span class="fu">278980255126953125e</span><span class="op">+</span>00<span class="op">,..</span> 3<span class="op">.</span><span class="fu">949900865554809570e</span><span class="op">+</span>00</span>
<span id="cb5-4"><a href="#cb5-4" aria-hidden="true" tabindex="-1"></a>1<span class="op">.</span><span class="fu">160453605651855469e</span><span class="op">+</span>01<span class="op">,</span>1<span class="op">.</span><span class="fu">707991600036621094e</span><span class="op">+</span>01<span class="op">,..</span> 1<span class="op">.</span><span class="fu">334794044494628906e</span><span class="op">+</span>00</span>
<span id="cb5-5"><a href="#cb5-5" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
<div class="sourceCode" id="cb6"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb6-1"><a href="#cb6-1" aria-hidden="true" tabindex="-1"></a><span class="co"># Example of csv file (uint8 number)</span></span>
<span id="cb6-2"><a href="#cb6-2" aria-hidden="true" tabindex="-1"></a><span class="co"># dtype=uint8</span></span>
<span id="cb6-3"><a href="#cb6-3" aria-hidden="true" tabindex="-1"></a>50<span class="op">,</span> 65<span class="op">,</span> 71<span class="op">,</span> 71</span>
<span id="cb6-4"><a href="#cb6-4" aria-hidden="true" tabindex="-1"></a>4<span class="op">.</span><span class="fu">800000000000000000e</span><span class="op">+</span>01<span class="op">,</span>6<span class="op">.</span><span class="fu">700000000000000000e</span><span class="op">+</span>01<span class="op">,</span>7<span class="op">.</span><span class="fu">300000000000000000e</span><span class="op">+</span>01<span class="op">,</span>6<span class="op">.</span><span class="fu">700000000000000000e</span><span class="op">+</span>01</span>
<span id="cb6-5"><a href="#cb6-5" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
<ul>
<li><p>lines in the <code>&#39;csv&#39;</code> file are always parsed as the 32b float, after they are converted to int8/uint8 type if requested.</p></li>
<li><p>for a model with multiple IOs, one csv file should be strictly provided by input (respectively by output). The order in the CLI is used to know the order, name of the file is not considered. For the <code>npz</code> file, the index <code>&lt;idx&gt;</code> in the key name will be used.</p>
<div class="sourceCode" id="cb7"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb7-1"><a href="#cb7-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>model_file<span class="op">&gt;</span> <span class="op">-</span>vi inputs_one<span class="op">.</span><span class="fu">csv</span> inputs_two<span class="op">.</span><span class="fu">csv</span> <span class="op">-</span>vo outputs<span class="op">.</span><span class="fu">csv</span></span></code></pre></div></li>
</ul>
<p>For information, when the user data set are loaded, type and shape are reported:</p>
<div class="sourceCode" id="cb8"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb8-1"><a href="#cb8-1" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb8-2"><a href="#cb8-2" aria-hidden="true" tabindex="-1"></a>Setting validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb8-3"><a href="#cb8-3" aria-hidden="true" tabindex="-1"></a>  loading file<span class="op">:</span> <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_io<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb8-4"><a href="#cb8-4" aria-hidden="true" tabindex="-1"></a>  I<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>10<span class="op">,</span> 49<span class="op">,</span> 40<span class="op">,</span> 1<span class="op">)/</span>uint8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">,</span> 255<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>127<span class="op">.</span><span class="fu">323</span><span class="op">,</span> 73<span class="op">.</span><span class="fu">590</span><span class="op">],</span> scale<span class="op">=</span>0<span class="op">.</span><span class="fu">10196070</span> zp<span class="op">=</span>0<span class="op">,</span> Reshape_1</span>
<span id="cb8-5"><a href="#cb8-5" aria-hidden="true" tabindex="-1"></a>  O<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>10<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 4<span class="op">)/</span>uint8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">,</span> 255<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>63<span class="op">.</span><span class="fu">925</span><span class="op">,</span> 90<span class="op">.</span><span class="fu">967</span><span class="op">],</span> scale<span class="op">=</span>0<span class="op">.</span><span class="fu">00390625</span> zp<span class="op">=</span>0<span class="op">,</span> nl_2_fmt_conv</span>
<span id="cb8-6"><a href="#cb8-6" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
<div class="Tips">
<p><strong>Tip</strong> — Generated output validation files from a previous “validate” command can be used as input files.</p>
</div>
</section>
<section id="ref_post_proc_support" class="level2">
<h2>Output validation files for post-processing</h2>
<p>Inputs and predicted values are saved in different files without modifications (type and shape are conserved). This allows to use a <a href="#ref_script_ex">post-processing script</a> to compute the user-defined metrics.</p>
<pre><code>&lt;output-directory-path&gt;\&lt;name&gt;_val_io.npz</code></pre>
<p>The <code>npz</code> file is a standard numpy binary format storing several arrays. The following key entries are used to store the data: <code>&#39;m_inputs_&lt;idx&gt;&#39;</code>, <code>&#39;c_inputs_&lt;idx&gt;&#39;</code>, <code>&#39;m_outputs_&lt;idx&gt;&#39;</code> and <code>&#39;c_outputs_&lt;idx&gt;&#39;</code>.</p>
<p>For quick and easy debug purpose, <code>&#39;csv&#39;</code> files (txt file) are also created by input and output. However, the number of samples and data by sample is voluntarily limited to limit the execution time. By default, only the 128 first samples with a size lower that 512 items are saved. If the limit is exceeded, file is created but without the data. <code>&#39;--save-csv&#39;</code> option can be used to force the creation of the <code>&#39;csv&#39;</code> files with all the data. The created <code>&#39;csv&#39;</code> file respects the format of the <a href="#ref_valio_arg">input validation files</a>. The name of file is defined as follow:</p>
<pre><code>&lt;network_name&gt;_[m,c]_[inputs,outputs]_&lt;idx&gt;.csv</code></pre>
<div class="sourceCode" id="cb11"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb11-1"><a href="#cb11-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>model_file<span class="op">&gt;</span> <span class="op">-</span>v 2 <span class="op">--</span>save<span class="op">-</span>csv</span>
<span id="cb11-2"><a href="#cb11-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb11-3"><a href="#cb11-3" aria-hidden="true" tabindex="-1"></a>Saving validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb11-4"><a href="#cb11-4" aria-hidden="true" tabindex="-1"></a> output directory<span class="op">:</span> <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span></span>
<span id="cb11-5"><a href="#cb11-5" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_io<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb11-6"><a href="#cb11-6" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_m_inputs_1<span class="op">.</span><span class="fu">csv</span></span>
<span id="cb11-7"><a href="#cb11-7" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_c_inputs_1<span class="op">.</span><span class="fu">csv</span></span>
<span id="cb11-8"><a href="#cb11-8" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_m_outputs_1<span class="op">.</span><span class="fu">csv</span></span>
<span id="cb11-9"><a href="#cb11-9" aria-hidden="true" tabindex="-1"></a> m_outputs_1<span class="op">:</span> <span class="op">(</span>10<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 4<span class="op">)/</span>uint8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">,</span> 255<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>63<span class="op">.</span><span class="fu">925</span><span class="op">,</span> 90<span class="op">.</span><span class="fu">967</span><span class="op">],</span></span>
<span id="cb11-10"><a href="#cb11-10" aria-hidden="true" tabindex="-1"></a>              nl_2_fmt_conv</span>
<span id="cb11-11"><a href="#cb11-11" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_c_outputs_1<span class="op">.</span><span class="fu">csv</span></span>
<span id="cb11-12"><a href="#cb11-12" aria-hidden="true" tabindex="-1"></a> c_outputs_1<span class="op">:</span> <span class="op">(</span>10<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 4<span class="op">)/</span>uint8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">,</span> 255<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>63<span class="op">.</span><span class="fu">925</span><span class="op">,</span> 90<span class="op">.</span><span class="fu">967</span><span class="op">],</span></span>
<span id="cb11-13"><a href="#cb11-13" aria-hidden="true" tabindex="-1"></a>              scale<span class="op">=</span>0<span class="op">.</span><span class="fu">00390625</span> zp<span class="op">=</span>0<span class="op">,</span> nl_2_fmt_conv</span>
<span id="cb11-14"><a href="#cb11-14" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
</section>
</section>
<section id="metrics" class="level1">
<h1>Metrics</h1>
<section id="ref_complexity" class="level2">
<h2>Computational complexity: MACC and cycles/MACC</h2>
<p>During the analyzing step of the model, global computational complexity is displayed and logged: <code>&#39;MACC&#39;</code> (refer to <a href="command_line_interface.html#analyze-command">[CLI], “Analyze command”</a> section). It indicates the number of multiply-and-accumulate operations which are requested to perform an inference. Value is computed independently of the data format (floating-point or integer) or the underlying C-implementation. As illustrated in the report (“graph” section, table form), value of the MACC can be refined layer-per-layer according the applied optimizations (fusing and/or folding processes).</p>
<p>Like this metric impacts directly, the latency of the application, on-device profiling is possible thanks to the <em>aiSystemPerformance</em> or <em>aiValidation</em> test applications <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">[UM]</a>. They permit to report the average number of CPU clocks requested for the whole C-model allowing to compute the number of requested <em>cycles by MACC</em>. This indicator highlights the <em>global efficiency</em> of the underlying C-implementation, including the HW platform setting aspects. With the validation on-device, the <a href="command_line_interface.html#validate-command">execution-time by layer</a> <a href="command_line_interface.html">[CLI]</a> is also evaluated.</p>
<figure>
<img src="" property="center" style="width:50.0%" />
</figure>
<div class="sourceCode" id="cb12"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb12-1"><a href="#cb12-1" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb12-2"><a href="#cb12-2" aria-hidden="true" tabindex="-1"></a>Results <span class="kw">for</span> 10 inference<span class="op">(</span>s<span class="op">)</span> <span class="op">-</span> average per inference</span>
<span id="cb12-3"><a href="#cb12-3" aria-hidden="true" tabindex="-1"></a> device              <span class="op">:</span> 0x431 <span class="op">-</span> STM32F411xC<span class="op">/</span>E @100<span class="op">/</span>100MHz fpu<span class="op">,</span>art_lat<span class="op">=</span>3<span class="op">,</span></span>
<span id="cb12-4"><a href="#cb12-4" aria-hidden="true" tabindex="-1"></a>                       art_prefetch<span class="op">,</span>art_icache<span class="op">,</span>art_dcache</span>
<span id="cb12-5"><a href="#cb12-5" aria-hidden="true" tabindex="-1"></a> duration            <span class="op">:</span> 0<span class="op">.</span><span class="fu">351ms</span></span>
<span id="cb12-6"><a href="#cb12-6" aria-hidden="true" tabindex="-1"></a> CPU cycles          <span class="op">:</span> 35062</span>
<span id="cb12-7"><a href="#cb12-7" aria-hidden="true" tabindex="-1"></a> cycles<span class="op">/</span>MACC         <span class="op">:</span> 8<span class="op">.</span><span class="fu">74</span></span>
<span id="cb12-8"><a href="#cb12-8" aria-hidden="true" tabindex="-1"></a> c_nodes             <span class="op">:</span> 6</span>
<span id="cb12-9"><a href="#cb12-9" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
<p>Be aware that no theoretical relation is defined between the reported complexity and the real performance of the implemented C-model, due to the variability of the targeted environments considering the used Arm® tool-chains, MCU and underlying sub-system memory settings, NN topologies and layers. It is difficult to provide off-line an accurate number of CPU cycles/MACC vs. a given STM32 system settings. However, out-of-the-box, the following rough estimations can be used for a 32-bit floating-point C-model.</p>
<table>
<thead>
<tr class="header">
<th style="text-align: left;">STM32 series based on</th>
<th style="text-align: left;">cycles/MACC</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">Arm® Cortex®-M4</td>
<td style="text-align: left;">~9</td>
</tr>
<tr class="even">
<td style="text-align: left;">Arm® Cortex®-M7</td>
<td style="text-align: left;">~6</td>
</tr>
</tbody>
</table>
<p>For the quantized models, this factor can be approximately divided by 2 on average.</p>
</section>
<section id="ref_memory_occupancy" class="level2">
<h2>Memory-related metrics</h2>
<section id="main-contributors" class="level3">
<h3>Main contributors</h3>
<p>To deploy a model in a resource-constrained runtime environment, the requested ROM and RAM sizes are the key factors. Two main memory contributors are directly reported through the fields: <code>&#39;weights (ro)&#39;</code> and <code>&#39;activations (rw)&#39;</code> (refer to <a href="command_line_interface.html#analyze-command">[CLI], “Analyze command”</a> section). The first (also called <code>ROM</code> or <code>FLASH</code>) designates the size in bytes requested to store the weights/bias or other constant params. They are generally placed in a read-only memory-mapped segment like the embedded STM32 flash memory (also defined as <code>.rodata</code> section). Second (also called <code>RAM</code> or activations buffer) designates the size in bytes requested to store the intermediate results (including optionally the buffers for the <a href="embedded_client_api.html#sec_alloc_inputs">input or/and output tensors</a> of the model). It can be considered as a private heap (or scratch buffer) only used by the C-runtime engine during an inference. It should be placed in a read-write memory-mapped segment like the embedded STM32 RAM memory (`.bss, .data or .heap sections).</p>
<div class="Tips">
<p><strong>Tip</strong> — <a href="embedded_client_api.html#sec_data_placement">“AI buffers and privileged placement”</a> section provides more details about the integration aspects.</p>
</div>
</section>
<section id="including-the-kernel-codedata-sizes" class="level3">
<h3>Including the kernel code/data sizes</h3>
<p>To have a complete view, by refining all AI contributors, the relocatable binary support (refer to <a href="relocatable.html">[RELOC]</a>) can be used. The generated report allows to have more details about the real requested memory usage to execute the AI stack. It includes also the requested size of the data and code sections for the used kernels (part of the STM32 network runtime library) and the size for the generated C-structure and parameters for a given model (part of the generated <code>network.c</code> file).</p>
<p>In the following table, only the AI contributors are considered for each code/data sections. <code>weights</code> section designates the section to store the weights/bias and <code>act. size</code> indicates the size of the activations buffer.</p>
<div class="sourceCode" id="cb13"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb13-1"><a href="#cb13-1" aria-hidden="true" tabindex="-1"></a>Runtime memory layout <span class="op">(</span>series<span class="op">=</span><span class="st">&quot;stm32f4&quot;</span><span class="op">)</span></span>
<span id="cb13-2"><a href="#cb13-2" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb13-3"><a href="#cb13-3" aria-hidden="true" tabindex="-1"></a>section      size <span class="op">(</span>bytes<span class="op">)</span></span>
<span id="cb13-4"><a href="#cb13-4" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb13-5"><a href="#cb13-5" aria-hidden="true" tabindex="-1"></a>header                100<span class="op">*</span></span>
<span id="cb13-6"><a href="#cb13-6" aria-hidden="true" tabindex="-1"></a>txt                 7<span class="op">,</span>864      network<span class="op">+</span>kernel</span>
<span id="cb13-7"><a href="#cb13-7" aria-hidden="true" tabindex="-1"></a>rodata                128      network<span class="op">+</span>kernel</span>
<span id="cb13-8"><a href="#cb13-8" aria-hidden="true" tabindex="-1"></a><span class="kw">data</span>                1<span class="op">,</span>756      network<span class="op">+</span>kernel</span>
<span id="cb13-9"><a href="#cb13-9" aria-hidden="true" tabindex="-1"></a>bss                   132      network<span class="op">+</span>kernel</span>
<span id="cb13-10"><a href="#cb13-10" aria-hidden="true" tabindex="-1"></a>got                   108<span class="op">*</span></span>
<span id="cb13-11"><a href="#cb13-11" aria-hidden="true" tabindex="-1"></a>rel                   504<span class="op">*</span></span>
<span id="cb13-12"><a href="#cb13-12" aria-hidden="true" tabindex="-1"></a>weights            15<span class="op">,</span>560      network</span>
<span id="cb13-13"><a href="#cb13-13" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb13-14"><a href="#cb13-14" aria-hidden="true" tabindex="-1"></a>FLASH size         25<span class="op">,</span>308 <span class="op">+</span> 712<span class="op">*</span> <span class="op">(+</span>2<span class="op">.</span><span class="fu">81</span><span class="op">%)</span></span>
<span id="cb13-15"><a href="#cb13-15" aria-hidden="true" tabindex="-1"></a>RAM size<span class="op">**</span>          1<span class="op">,</span>888 <span class="op">+</span> 108<span class="op">*</span> <span class="op">(+</span>5<span class="op">.</span><span class="fu">72</span><span class="op">%)</span></span>
<span id="cb13-16"><a href="#cb13-16" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb13-17"><a href="#cb13-17" aria-hidden="true" tabindex="-1"></a>bin size           26<span class="op">,</span>024      binary image</span>
<span id="cb13-18"><a href="#cb13-18" aria-hidden="true" tabindex="-1"></a>act<span class="op">.</span> size             192      activations buffer</span>
<span id="cb13-19"><a href="#cb13-19" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb13-20"><a href="#cb13-20" aria-hidden="true" tabindex="-1"></a><span class="op">(*)</span> extra bytes <span class="kw">for</span> relocatable support</span>
<span id="cb13-21"><a href="#cb13-21" aria-hidden="true" tabindex="-1"></a><span class="op">(**)</span> RAM <span class="kw">for</span> IO should be added <span class="kw">if</span> not allocated <span class="kw">in</span> activations buffer</span></code></pre></div>
</section>
<section id="system-heap-and-stack-size" class="level3">
<h3>System heap and stack size</h3>
<p>By construction, the kernels are designed to not use the system heap (no explicit call to <code>malloc/free</code> like functions). If a minimal stack is requested to use the runtime, the <em>aiSystemPerformance</em> test application has been designed to know the requested stack size.</p>
</section>
</section>
<section id="ref_acc" class="level2">
<h2>Classification accuracy (acc)</h2>
<p>For classifier model type, <em>Classification accuracy</em> is what we usually mean, when the term <em>accuracy</em> is used. <em>ACC</em> is the ratio between of correct predictions to the total number of inputs. This indicator evaluates the performance of the <em>classifier</em> model, if a <em>regressor</em> type is passed, the <em>ACC</em> is <strong>NOT</strong> calculated and <code>n.a.</code> value is reported.</p>
<figure>
<img src="" property="center" style="width:40.0%" />
</figure>
<div class="Note">
<p><strong>Info</strong> — No threshold is applied to determine if a given class is detected or not. Only the maximum value along the axis is used.</p>
</div>
<div class="sourceCode" id="cb14"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb14-1"><a href="#cb14-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb14-2"><a href="#cb14-2" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> accuracy_score</span>
<span id="cb14-3"><a href="#cb14-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb14-4"><a href="#cb14-4" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> acc(ref, pred):</span>
<span id="cb14-5"><a href="#cb14-5" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Classification accuracy (ACC).&quot;&quot;&quot;</span></span>
<span id="cb14-6"><a href="#cb14-6" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> accuracy_score(np.argmax(ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(pred, axis<span class="op">=</span><span class="dv">1</span>))</span></code></pre></div>
</section>
<section id="ref_rmse" class="level2">
<h2>Root Mean Square Error (rmse)</h2>
<p><em>RMSE</em> is quite similar to <em>MAE</em>, the only difference being that <em>RMSE</em> takes the average of the square of the difference between the original values and the predicted values. As, we take square of the error, the effect of larger errors become more pronounced then smaller error, hence the model can now focus more on the larger errors. <em>RMSE</em> is computed for the flattened array (element-wise along the array), returning a scalar value. If <strong>[ R ]</strong> is not provided <strong>[ R’ ]</strong> is used.</p>
<figure>
<img src="" property="center" style="width:30.0%" />
</figure>
<div class="sourceCode" id="cb15"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb15-1"><a href="#cb15-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb15-2"><a href="#cb15-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb15-3"><a href="#cb15-3" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> rmse(ref, pred):</span>
<span id="cb15-4"><a href="#cb15-4" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return Root Mean Squared Error (RMSE).&quot;&quot;&quot;</span></span>
<span id="cb15-5"><a href="#cb15-5" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> np.sqrt(((ref <span class="op">-</span> pred).astype(np.float64) <span class="op">**</span> <span class="dv">2</span>).mean())</span></code></pre></div>
</section>
<section id="ref_mae" class="level2">
<h2>Mean Absolute Error (mae)</h2>
<p><em>MAE</em> is the average of the difference between the original value (or reference value, Rj) and the predicted value Pj. It gives the measure of how far the predictions were from the actual output. However, they don’t gives any idea of the direction of the error i.e. whether we are under predicting the data or over predicting the data. <em>MAE</em> is computed for the flattened array (element-wise along the array), returning a scalar value. If <strong>[ R ]</strong> is not provided <strong>[ R’ ]</strong> is used.</p>
<figure>
<img src="" property="center" style="width:25.0%" />
</figure>
<div class="sourceCode" id="cb16"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb16-1"><a href="#cb16-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb16-2"><a href="#cb16-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb16-3"><a href="#cb16-3" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> mae(ref, pred):</span>
<span id="cb16-4"><a href="#cb16-4" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return Mean Absolute Error (MAE).&quot;&quot;&quot;</span></span>
<span id="cb16-5"><a href="#cb16-5" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> (np.<span class="bu">abs</span>(ref <span class="op">-</span> pred).astype(np.float64)).mean()</span></code></pre></div>
</section>
<section id="ref_l2r" class="level2">
<h2>L2 relative error (l2r)</h2>
<p><em>L2r</em> is the scalar value of the relative 2-norm or Euclidean distance between the generated values of the original model <strong>[ R’ ]</strong> and the C-model <strong>[ P ]</strong>. When possible, this metric is also reported for the output of a C-layer matching with the original layer (see <a href="command_line_interface.html#ref_complexity_per_layer">[CLI] “Extended complexity report per layer”</a> section).</p>
<figure>
<img src="" property="center" style="width:35.0%" />
</figure>
<div class="sourceCode" id="cb17"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb17-1"><a href="#cb17-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb17-2"><a href="#cb17-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb17-3"><a href="#cb17-3" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> l2r(ref, pred):</span>
<span id="cb17-4"><a href="#cb17-4" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Compute L2 relative error&quot;&quot;&quot;</span></span>
<span id="cb17-5"><a href="#cb17-5" aria-hidden="true" tabindex="-1"></a>  <span class="kw">def</span> magnitude(v):</span>
<span id="cb17-6"><a href="#cb17-6" aria-hidden="true" tabindex="-1"></a>    <span class="cf">return</span> np.sqrt(np.<span class="bu">sum</span>(np.square(v).flatten()))</span>
<span id="cb17-7"><a href="#cb17-7" aria-hidden="true" tabindex="-1"></a>  mag <span class="op">=</span> magnitude(pred) <span class="op">+</span> np.finfo(np.float32).eps</span>
<span id="cb17-8"><a href="#cb17-8" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> magnitude(ref <span class="op">-</span> pred) <span class="op">/</span> mag</span></code></pre></div>
</section>
<section id="ref_mean" class="level2">
<h2>Arithmetic mean of the error (mean)</h2>
<p><em>mean</em> is the scalar value of the arithmetic mean between the original value (or reference value, Rj) and the predicted value Pj. It is also called <em>bias</em>.</p>
<div class="sourceCode" id="cb18"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb18-1"><a href="#cb18-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb18-2"><a href="#cb18-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb18-3"><a href="#cb18-3" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> mean(ref, pred):</span>
<span id="cb18-4"><a href="#cb18-4" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return the Arithmetic Mean (MEAN).&quot;&quot;&quot;</span></span>
<span id="cb18-5"><a href="#cb18-5" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> np.mean(ref <span class="op">-</span> pred)</span></code></pre></div>
</section>
<section id="ref_std" class="level2">
<h2>Standard deviation of the error (std)</h2>
<p><em>std</em> is the scalar value of the standard deviation between the original value (or reference value, Rj) and the predicted value Pj. It provides a measure of the spread of a distribution of the error.</p>
<div class="sourceCode" id="cb19"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb19-1"><a href="#cb19-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb19-2"><a href="#cb19-2" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb19-3"><a href="#cb19-3" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> mean(ref, pred):</span>
<span id="cb19-4"><a href="#cb19-4" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return the Arithmetic Mean (MEAN).&quot;&quot;&quot;</span></span>
<span id="cb19-5"><a href="#cb19-5" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> np.std(ref <span class="op">-</span> pred)</span></code></pre></div>
</section>
<section id="ref_cm" class="level2">
<h2>Confusion matrix (CM)</h2>
<p>When outputs are provided and the model is considered as a <a href="#ref_regressor_model">classifier</a>, a confusion matrix is displayed and logged for the C-model and the reference model. It describes the complete performance of the model. Note that if a <em>regressor</em> type is passed, the confusion matrix is <strong>NOT</strong> calculated only the <a href="#ref_rmse">RMSE</a>, <a href="#ref_mae">MAE</a> and <a href="#ref_l2r">L2r</a> metrics are computed.</p>
<div class="sourceCode" id="cb20"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb20-1"><a href="#cb20-1" aria-hidden="true" tabindex="-1"></a>8 classes <span class="op">(</span>50 samples<span class="op">)</span></span>
<span id="cb20-2"><a href="#cb20-2" aria-hidden="true" tabindex="-1"></a><span class="op">------------------------------------------------</span></span>
<span id="cb20-3"><a href="#cb20-3" aria-hidden="true" tabindex="-1"></a>C0         4    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb20-4"><a href="#cb20-4" aria-hidden="true" tabindex="-1"></a>C1         <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb20-5"><a href="#cb20-5" aria-hidden="true" tabindex="-1"></a>C2         <span class="op">.</span>    <span class="op">.</span>    6    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb20-6"><a href="#cb20-6" aria-hidden="true" tabindex="-1"></a>C3         <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb20-7"><a href="#cb20-7" aria-hidden="true" tabindex="-1"></a>C4         <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    5    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb20-8"><a href="#cb20-8" aria-hidden="true" tabindex="-1"></a>C5         <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    6    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb20-9"><a href="#cb20-9" aria-hidden="true" tabindex="-1"></a>C6         <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span></span>
<span id="cb20-10"><a href="#cb20-10" aria-hidden="true" tabindex="-1"></a>C7         <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    6</span></code></pre></div>
<p><strong><em>X-Cross</em></strong> confusion matrix or accuracy uses the outputs of the reference model to build the ground truth values. They are used to evaluate and to compare the performance of the C-model.</p>
<div class="HTips">
<p><strong>Note</strong> — The confusion matrix is only displayed when the number of class is lower or equal to 20.</p>
</div>
</section>
</section>
<section id="metric_interpretation" class="level1">
<h1>Interpretation of the results</h1>
<p>This section provided typical examples to interpret the reported metrics. It is <strong>important</strong> to keep in mind the following points:</p>
<ul>
<li>proposed built-in validation flow is mainly based on the comparison of output predictions generated by the c-model (X86 or/and STM32 C-runtime) and the values predict by the origin inference engine (i.e. ONNX runtime, TFLite interpreter and Keras predict method). Consequently, the precision is highly dependent on the underlying implementation, and to the way to accumulate and round the intermediate or final results.
<ul>
<li><em>For more accurate results, we recommend to consider a validation flow using pre and post processing and representative validation data set on the top of the X-CUBE-AI network runtime library (X86 or/and STM32 C-runtime) (refer to <a href="how_to_run_a_model_locally.html">[C-RUN] “How to run locally a c-model”</a> article)</em></li>
</ul></li>
<li>usage of the flattened random data with a given range for the input samples is not representative of real distribution used to train the original model. Distribution between the channels is always considered as uniform.
<ul>
<li><em>We recommend to provide the <a href="#ref_data_recom">representative pre-processed data</a> to apply the built-in validation flow.</em></li>
</ul></li>
</ul>
<blockquote>
<p>Most part of the used Python code illustrating this example is available inside the X-CUBE-AI pack: <code>%X_CUBE_AI_DIR%/scripts/ai_runner/example/mnist</code> directory (refer to <a href="setting_env.html">[INST]</a> article)</p>
</blockquote>
<section id="floating-point-model" class="level2">
<h2>Floating point model</h2>
<section id="with-random-data" class="level3">
<h3>with random data</h3>
<p>This section illustrates a typical example where a 32b floating point model is validated with the random data. Default range <code>[0, 1.0]</code> is used here, because the image data set has been normalized between 0 and 1.</p>
<ul>
<li>X-cross #1 results highlights that the predicted values of both models are close.</li>
<li>X-Cross #1 <code>l2r=0.000001551</code> illustrates that with the random inputs, the predicted values of the c-model are identical to the outputs generated by the Keras model interpreter.</li>
<li><code>acc=100%</code> metric indicates that a given input sample is classified with the same way in both cases.</li>
</ul>
<div class="sourceCode" id="cb21"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb21-1"><a href="#cb21-1" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> tensorflow <span class="im">as</span> tf</span>
<span id="cb21-2"><a href="#cb21-2" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb21-3"><a href="#cb21-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-4"><a href="#cb21-4" aria-hidden="true" tabindex="-1"></a>H, W, C <span class="op">=</span> <span class="dv">28</span>, <span class="dv">28</span>, <span class="dv">1</span></span>
<span id="cb21-5"><a href="#cb21-5" aria-hidden="true" tabindex="-1"></a>IN_SHAPE <span class="op">=</span> (H, W, C)</span>
<span id="cb21-6"><a href="#cb21-6" aria-hidden="true" tabindex="-1"></a>NB_CLASSES <span class="op">=</span> <span class="dv">10</span></span>
<span id="cb21-7"><a href="#cb21-7" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-8"><a href="#cb21-8" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> load_data_set():</span>
<span id="cb21-9"><a href="#cb21-9" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Load the data&quot;&quot;&quot;</span></span>
<span id="cb21-10"><a href="#cb21-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-11"><a href="#cb21-11" aria-hidden="true" tabindex="-1"></a>  mnist <span class="op">=</span> tf.keras.datasets.mnist</span>
<span id="cb21-12"><a href="#cb21-12" aria-hidden="true" tabindex="-1"></a>  (train_images, train_labels), (test_images, test_labels) <span class="op">=</span> mnist.load_data()</span>
<span id="cb21-13"><a href="#cb21-13" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-14"><a href="#cb21-14" aria-hidden="true" tabindex="-1"></a>  <span class="co"># Normalize the input image so that each pixel value is between 0 to 1.</span></span>
<span id="cb21-15"><a href="#cb21-15" aria-hidden="true" tabindex="-1"></a>  x_train <span class="op">=</span> x_train <span class="op">/</span> <span class="fl">255.0</span></span>
<span id="cb21-16"><a href="#cb21-16" aria-hidden="true" tabindex="-1"></a>  x_test <span class="op">=</span> x_test <span class="op">/</span> <span class="fl">255.0</span></span>
<span id="cb21-17"><a href="#cb21-17" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-18"><a href="#cb21-18" aria-hidden="true" tabindex="-1"></a>  x_train <span class="op">=</span> x_train.reshape(x_train.shape[<span class="dv">0</span>], H, W, C).astype(np.float32)</span>
<span id="cb21-19"><a href="#cb21-19" aria-hidden="true" tabindex="-1"></a>  x_test <span class="op">=</span> x_test.reshape(x_test.shape[<span class="dv">0</span>], H, W, C).astype(np.float32)</span>
<span id="cb21-20"><a href="#cb21-20" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-21"><a href="#cb21-21" aria-hidden="true" tabindex="-1"></a>  <span class="co"># convert class vectors to binary class matrices</span></span>
<span id="cb21-22"><a href="#cb21-22" aria-hidden="true" tabindex="-1"></a>  y_train <span class="op">=</span> tf.keras.utils.to_categorical(y_train, NB_CLASSES)</span>
<span id="cb21-23"><a href="#cb21-23" aria-hidden="true" tabindex="-1"></a>  y_test <span class="op">=</span> tf.keras.utils.to_categorical(y_test, NB_CLASSES)</span>
<span id="cb21-24"><a href="#cb21-24" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-25"><a href="#cb21-25" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> x_train, y_train, x_test, y_test</span>
<span id="cb21-26"><a href="#cb21-26" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-27"><a href="#cb21-27" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> build_model():</span>
<span id="cb21-28"><a href="#cb21-28" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Define the model&quot;&quot;&quot;</span></span>
<span id="cb21-29"><a href="#cb21-29" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-30"><a href="#cb21-30" aria-hidden="true" tabindex="-1"></a>  model <span class="op">=</span> tf.keras.Sequential([</span>
<span id="cb21-31"><a href="#cb21-31" aria-hidden="true" tabindex="-1"></a>    tf.keras.layers.InputLayer(input_shape<span class="op">=</span>IN_SHAPE),</span>
<span id="cb21-32"><a href="#cb21-32" aria-hidden="true" tabindex="-1"></a>    tf.keras.layers.Conv2D(filters<span class="op">=</span><span class="dv">12</span>, kernel_size<span class="op">=</span>(<span class="dv">3</span>, <span class="dv">3</span>), activation<span class="op">=</span>tf.nn.relu),</span>
<span id="cb21-33"><a href="#cb21-33" aria-hidden="true" tabindex="-1"></a>    tf.keras.layers.MaxPooling2D(pool_size<span class="op">=</span>(<span class="dv">2</span>, <span class="dv">2</span>)),</span>
<span id="cb21-34"><a href="#cb21-34" aria-hidden="true" tabindex="-1"></a>    tf.keras.layers.Dropout(<span class="fl">0.5</span>),</span>
<span id="cb21-35"><a href="#cb21-35" aria-hidden="true" tabindex="-1"></a>    tf.keras.layers.Flatten(),</span>
<span id="cb21-36"><a href="#cb21-36" aria-hidden="true" tabindex="-1"></a>    tf.keras.layers.Dense(<span class="dv">10</span>, activation<span class="op">=</span><span class="st">&#39;softmax&#39;</span>)</span>
<span id="cb21-37"><a href="#cb21-37" aria-hidden="true" tabindex="-1"></a>  ])</span>
<span id="cb21-38"><a href="#cb21-38" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb21-39"><a href="#cb21-39" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> model</span></code></pre></div>
<div class="sourceCode" id="cb22"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb22-1"><a href="#cb22-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>model_fp32<span class="op">&gt;</span></span>
<span id="cb22-2"><a href="#cb22-2" aria-hidden="true" tabindex="-1"></a>Neural Network Tools <span class="kw">for</span> STM32AI v1<span class="op">.</span><span class="fu">5</span><span class="op">.</span><span class="fu">1</span> <span class="op">(</span>STM<span class="op">.</span><span class="fu">ai</span> v7<span class="op">.</span><span class="fu">0</span><span class="op">.</span><span class="fu">0</span><span class="op">)</span></span>
<span id="cb22-3"><a href="#cb22-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-4"><a href="#cb22-4" aria-hidden="true" tabindex="-1"></a>Setting validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb22-5"><a href="#cb22-5" aria-hidden="true" tabindex="-1"></a> generating random <span class="kw">data</span><span class="op">,</span> size<span class="op">=</span>10<span class="op">,</span> seed<span class="op">=</span>42<span class="op">,</span> range<span class="op">=</span>default</span>
<span id="cb22-6"><a href="#cb22-6" aria-hidden="true" tabindex="-1"></a> I<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>10<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 1<span class="op">.</span><span class="fu">000</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">495</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">289</span><span class="op">],</span> input_0</span>
<span id="cb22-7"><a href="#cb22-7" aria-hidden="true" tabindex="-1"></a> No output<span class="op">/</span>reference samples are provided</span>
<span id="cb22-8"><a href="#cb22-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-9"><a href="#cb22-9" aria-hidden="true" tabindex="-1"></a>Running the STM AI c<span class="op">-</span>model <span class="op">(</span>AI RUNNER<span class="op">)...(</span>name<span class="op">=</span>network<span class="op">,</span> mode<span class="op">=</span>x86<span class="op">)..</span></span>
<span id="cb22-10"><a href="#cb22-10" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb22-11"><a href="#cb22-11" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-12"><a href="#cb22-12" aria-hidden="true" tabindex="-1"></a>Running the Keras model<span class="op">...</span></span>
<span id="cb22-13"><a href="#cb22-13" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-14"><a href="#cb22-14" aria-hidden="true" tabindex="-1"></a>Saving validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb22-15"><a href="#cb22-15" aria-hidden="true" tabindex="-1"></a> output directory<span class="op">:</span> <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span></span>
<span id="cb22-16"><a href="#cb22-16" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_io<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb22-17"><a href="#cb22-17" aria-hidden="true" tabindex="-1"></a> m_outputs_1<span class="op">:</span> <span class="op">(</span>10<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">975</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">100</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">253</span><span class="op">],</span> dense_nl</span>
<span id="cb22-18"><a href="#cb22-18" aria-hidden="true" tabindex="-1"></a> c_outputs_1<span class="op">:</span> <span class="op">(</span>10<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">975</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">100</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">253</span><span class="op">],</span> dense_nl</span>
<span id="cb22-19"><a href="#cb22-19" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-20"><a href="#cb22-20" aria-hidden="true" tabindex="-1"></a>Computing the metrics<span class="op">...</span></span>
<span id="cb22-21"><a href="#cb22-21" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-22"><a href="#cb22-22" aria-hidden="true" tabindex="-1"></a> Cross accuracy report <span class="co">#1 (reference vs C-model)</span></span>
<span id="cb22-23"><a href="#cb22-23" aria-hidden="true" tabindex="-1"></a> <span class="op">----------------------------------------------------------------------------------------------------</span></span>
<span id="cb22-24"><a href="#cb22-24" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> the output of the reference model is used as ground truth<span class="op">/</span>reference value</span>
<span id="cb22-25"><a href="#cb22-25" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 10 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb22-26"><a href="#cb22-26" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-27"><a href="#cb22-27" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>100<span class="op">.</span><span class="fu">00</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">000000422</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">000000162</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">000001551</span></span>
<span id="cb22-28"><a href="#cb22-28" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-29"><a href="#cb22-29" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>10 samples<span class="op">)</span></span>
<span id="cb22-30"><a href="#cb22-30" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb22-31"><a href="#cb22-31" aria-hidden="true" tabindex="-1"></a>  C0        0    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-32"><a href="#cb22-32" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>    0    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-33"><a href="#cb22-33" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-34"><a href="#cb22-34" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    0    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-35"><a href="#cb22-35" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    0    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-36"><a href="#cb22-36" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-37"><a href="#cb22-37" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    0    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-38"><a href="#cb22-38" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    0    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb22-39"><a href="#cb22-39" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    0    <span class="op">.</span></span>
<span id="cb22-40"><a href="#cb22-40" aria-hidden="true" tabindex="-1"></a>  C9        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    0</span>
<span id="cb22-41"><a href="#cb22-41" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-42"><a href="#cb22-42" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-43"><a href="#cb22-43" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb22-44"><a href="#cb22-44" aria-hidden="true" tabindex="-1"></a><span class="op">----------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">-------------------------------</span></span>
<span id="cb22-45"><a href="#cb22-45" aria-hidden="true" tabindex="-1"></a>Output       acc       rmse            l2r             tensor</span>
<span id="cb22-46"><a href="#cb22-46" aria-hidden="true" tabindex="-1"></a><span class="op">----------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">-------------------------------</span></span>
<span id="cb22-47"><a href="#cb22-47" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1   100.00%   0.000000422 ... 0.000001551 ...  dense_nl, ai_float,...</span></span>
<span id="cb22-48"><a href="#cb22-48" aria-hidden="true" tabindex="-1"></a><span class="op">----------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">-------------------------------</span></span>
<span id="cb22-49"><a href="#cb22-49" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb22-50"><a href="#cb22-50" aria-hidden="true" tabindex="-1"></a>Creating report file <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_validate_report<span class="op">.</span><span class="fu">txt</span></span></code></pre></div>
</section>
<section id="with-representative-data-set" class="level3">
<h3>with representative data set</h3>
<p>When a representative user data set is used (including the ground truth values), the <code>ACC/CM</code> metrics are also evaluated with the original and generated c-models.</p>
<ul>
<li>X-cross #1 results highlights that the predicted values of both models are close.</li>
<li>X-Cross #1 <code>l2r=0.000000249</code> for example, indicates that with the random inputs, the predicted values of the c-model are identical to the outputs generated by the Keras model interpreter.</li>
<li>However, <em>x86 C-model #1</em> and <em>original model #1</em> lines indicate the relatively important <code>rmse/mae/l2r</code> errors. They are due here to the encoding of the provided ground truth values, one-hot encoded: <code>[0.0 0.0 1.0 .. 0.0]</code>. These errors could be more huge, if <a href="#ref_no_softmax">no <em>softmax</em> layer</a> is present.</li>
</ul>
<div class="sourceCode" id="cb23"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb23-1"><a href="#cb23-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>model_fp32<span class="op">&gt;</span> <span class="op">-</span>vi <span class="op">&lt;</span>data_directory<span class="op">&gt;/</span>data_reduced_test<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb23-2"><a href="#cb23-2" aria-hidden="true" tabindex="-1"></a>Neural Network Tools <span class="kw">for</span> STM32AI v1<span class="op">.</span><span class="fu">5</span><span class="op">.</span><span class="fu">1</span> <span class="op">(</span>STM<span class="op">.</span><span class="fu">ai</span> v7<span class="op">.</span><span class="fu">0</span><span class="op">.</span><span class="fu">0</span><span class="op">)</span></span>
<span id="cb23-3"><a href="#cb23-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-4"><a href="#cb23-4" aria-hidden="true" tabindex="-1"></a>Setting validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb23-5"><a href="#cb23-5" aria-hidden="true" tabindex="-1"></a> loading file<span class="op">:</span> <span class="op">&lt;</span>data_directory<span class="op">&gt;</span>\data_reduced_test<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb23-6"><a href="#cb23-6" aria-hidden="true" tabindex="-1"></a> <span class="op">-</span> samples are reshaped<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)</span> <span class="op">-&gt;</span> <span class="op">(</span>128<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)</span></span>
<span id="cb23-7"><a href="#cb23-7" aria-hidden="true" tabindex="-1"></a> <span class="op">-</span> samples are reshaped<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 10<span class="op">)</span> <span class="op">-&gt;</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)</span></span>
<span id="cb23-8"><a href="#cb23-8" aria-hidden="true" tabindex="-1"></a> I<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>128<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 1<span class="op">.</span><span class="fu">000</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">139</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">318</span><span class="op">],</span> input_0</span>
<span id="cb23-9"><a href="#cb23-9" aria-hidden="true" tabindex="-1"></a> O<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 1<span class="op">.</span><span class="fu">000</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">100</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">300</span><span class="op">],</span> dense_nl</span>
<span id="cb23-10"><a href="#cb23-10" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-11"><a href="#cb23-11" aria-hidden="true" tabindex="-1"></a>Running the STM AI c<span class="op">-</span>model <span class="op">(</span>AI RUNNER<span class="op">)...(</span>name<span class="op">=</span>network<span class="op">,</span> mode<span class="op">=</span>x86<span class="op">)</span></span>
<span id="cb23-12"><a href="#cb23-12" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb23-13"><a href="#cb23-13" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-14"><a href="#cb23-14" aria-hidden="true" tabindex="-1"></a>Running the Keras model<span class="op">...</span></span>
<span id="cb23-15"><a href="#cb23-15" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-16"><a href="#cb23-16" aria-hidden="true" tabindex="-1"></a>Saving validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb23-17"><a href="#cb23-17" aria-hidden="true" tabindex="-1"></a> output directory<span class="op">:&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span></span>
<span id="cb23-18"><a href="#cb23-18" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_io<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb23-19"><a href="#cb23-19" aria-hidden="true" tabindex="-1"></a> m_outputs_1<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 1<span class="op">.</span><span class="fu">000</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">100</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">288</span><span class="op">],</span> dense_nl</span>
<span id="cb23-20"><a href="#cb23-20" aria-hidden="true" tabindex="-1"></a> c_outputs_1<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 1<span class="op">.</span><span class="fu">000</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">100</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">288</span><span class="op">],</span> dense_nl</span>
<span id="cb23-21"><a href="#cb23-21" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-22"><a href="#cb23-22" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-23"><a href="#cb23-23" aria-hidden="true" tabindex="-1"></a>Computing the metrics<span class="op">...</span></span>
<span id="cb23-24"><a href="#cb23-24" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-25"><a href="#cb23-25" aria-hidden="true" tabindex="-1"></a> Accuracy report <span class="co">#1 for the generated x86 C-model</span></span>
<span id="cb23-26"><a href="#cb23-26" aria-hidden="true" tabindex="-1"></a> <span class="op">---------------------------------------------------------------------------------------------</span></span>
<span id="cb23-27"><a href="#cb23-27" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> computed against the provided ground truth values</span>
<span id="cb23-28"><a href="#cb23-28" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 128 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb23-29"><a href="#cb23-29" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-30"><a href="#cb23-30" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>97<span class="op">.</span><span class="fu">66</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">054929093</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">010250041</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">180345476</span></span>
<span id="cb23-31"><a href="#cb23-31" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-32"><a href="#cb23-32" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>128 samples<span class="op">)</span></span>
<span id="cb23-33"><a href="#cb23-33" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb23-34"><a href="#cb23-34" aria-hidden="true" tabindex="-1"></a>  C0       11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-35"><a href="#cb23-35" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>   19    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-36"><a href="#cb23-36" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-37"><a href="#cb23-37" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-38"><a href="#cb23-38" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   15    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-39"><a href="#cb23-39" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-40"><a href="#cb23-40" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-41"><a href="#cb23-41" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-42"><a href="#cb23-42" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   17    <span class="op">.</span></span>
<span id="cb23-43"><a href="#cb23-43" aria-hidden="true" tabindex="-1"></a>  C9        1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10</span>
<span id="cb23-44"><a href="#cb23-44" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-45"><a href="#cb23-45" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-46"><a href="#cb23-46" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-47"><a href="#cb23-47" aria-hidden="true" tabindex="-1"></a> Accuracy report <span class="co">#1 for the reference model</span></span>
<span id="cb23-48"><a href="#cb23-48" aria-hidden="true" tabindex="-1"></a> <span class="op">---------------------------------------------------------------------------------------------</span></span>
<span id="cb23-49"><a href="#cb23-49" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> computed against the provided ground truth values</span>
<span id="cb23-50"><a href="#cb23-50" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 128 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb23-51"><a href="#cb23-51" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-52"><a href="#cb23-52" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>97<span class="op">.</span><span class="fu">66</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">054929089</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">010250039</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">180345476</span></span>
<span id="cb23-53"><a href="#cb23-53" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-54"><a href="#cb23-54" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>128 samples<span class="op">)</span></span>
<span id="cb23-55"><a href="#cb23-55" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb23-56"><a href="#cb23-56" aria-hidden="true" tabindex="-1"></a>  C0       11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-57"><a href="#cb23-57" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>   19    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-58"><a href="#cb23-58" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-59"><a href="#cb23-59" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-60"><a href="#cb23-60" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   15    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-61"><a href="#cb23-61" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-62"><a href="#cb23-62" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-63"><a href="#cb23-63" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-64"><a href="#cb23-64" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   17    <span class="op">.</span></span>
<span id="cb23-65"><a href="#cb23-65" aria-hidden="true" tabindex="-1"></a>  C9        1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10</span>
<span id="cb23-66"><a href="#cb23-66" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-67"><a href="#cb23-67" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-68"><a href="#cb23-68" aria-hidden="true" tabindex="-1"></a> Cross accuracy report <span class="co">#1 (reference vs C-model)</span></span>
<span id="cb23-69"><a href="#cb23-69" aria-hidden="true" tabindex="-1"></a> <span class="op">---------------------------------------------------------------------------------------------</span></span>
<span id="cb23-70"><a href="#cb23-70" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> the output of the reference model is used as ground truth<span class="op">/</span>reference value</span>
<span id="cb23-71"><a href="#cb23-71" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 128 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb23-72"><a href="#cb23-72" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-73"><a href="#cb23-73" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>100<span class="op">.</span><span class="fu">00</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">000000076</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">000000016</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">000000249</span></span>
<span id="cb23-74"><a href="#cb23-74" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-75"><a href="#cb23-75" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>128 samples<span class="op">)</span></span>
<span id="cb23-76"><a href="#cb23-76" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb23-77"><a href="#cb23-77" aria-hidden="true" tabindex="-1"></a>  C0       12    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-78"><a href="#cb23-78" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>   19    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-79"><a href="#cb23-79" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-80"><a href="#cb23-80" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   12    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-81"><a href="#cb23-81" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-82"><a href="#cb23-82" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-83"><a href="#cb23-83" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-84"><a href="#cb23-84" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb23-85"><a href="#cb23-85" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   17    <span class="op">.</span></span>
<span id="cb23-86"><a href="#cb23-86" aria-hidden="true" tabindex="-1"></a>  C9        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10</span>
<span id="cb23-87"><a href="#cb23-87" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-88"><a href="#cb23-88" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-89"><a href="#cb23-89" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb23-90"><a href="#cb23-90" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">------------------------</span></span>
<span id="cb23-91"><a href="#cb23-91" aria-hidden="true" tabindex="-1"></a>Output              acc       rmse             l2r            tensor</span>
<span id="cb23-92"><a href="#cb23-92" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">------------------------</span></span>
<span id="cb23-93"><a href="#cb23-93" aria-hidden="true" tabindex="-1"></a>x86 c<span class="op">-</span>model <span class="co">#1      97.66%    0.054929093 ... 0.180345476 ... nl_4_fmt_conv, ai_i8,...</span></span>
<span id="cb23-94"><a href="#cb23-94" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   97.66%    0.054929089 ... 0.180345476 ... nl_4_fmt_conv, ai_i8,...</span></span>
<span id="cb23-95"><a href="#cb23-95" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          100.00%   0.000000076 ... 0.000000249 ... nl_4_fmt_conv, ai_i8,...</span></span>
<span id="cb23-96"><a href="#cb23-96" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">------------------------</span></span>
<span id="cb23-97"><a href="#cb23-97" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb23-98"><a href="#cb23-98" aria-hidden="true" tabindex="-1"></a>Creating report file <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_validate_report<span class="op">.</span><span class="fu">txt</span></span></code></pre></div>
</section>
<section id="ref_no_softmax" class="level3">
<h3>without softmax layer</h3>
<p>In the case where the last layer is not a <em>softmax</em> operator, the <em>x86 C-model #1</em> and <em>original model #1</em> results can indicate the huge <code>rmse/mae/l2r</code> errors (for example <code>~6.6+</code> here with another MNIST model). These errors are fully dependent of the values of the provided data. In this example, there are one-hot encoded: <code>[0.0 0.0 1.0 .. 0.0]</code>. Consequently, they are directly compared to the predicted values accumulating the significant differences. In this context, <code>acc</code> and *X-cross&quot; results are the more significant metrics. Note that the <code>--classifier</code> option is used to force the computation of the <code>ACC/CM</code> metrics.</p>
<div class="sourceCode" id="cb24"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb24-1"><a href="#cb24-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>float_model_wo_softmax<span class="op">&gt;</span> <span class="op">-</span>vi mnist_reduced_test<span class="op">.</span><span class="fu">npz</span> <span class="op">--</span>classifier</span>
<span id="cb24-2"><a href="#cb24-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb24-3"><a href="#cb24-3" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb24-4"><a href="#cb24-4" aria-hidden="true" tabindex="-1"></a><span class="op">---------------------------------------------------------------------------------------------------</span></span>
<span id="cb24-5"><a href="#cb24-5" aria-hidden="true" tabindex="-1"></a>Mode                acc       rmse          mae           l2r           tensor</span>
<span id="cb24-6"><a href="#cb24-6" aria-hidden="true" tabindex="-1"></a><span class="op">---------------------------------------------------------------------------------------------------</span></span>
<span id="cb24-7"><a href="#cb24-7" aria-hidden="true" tabindex="-1"></a>x86 C<span class="op">-</span>model <span class="co">#1      97.66%    6.643473148   5.626701355   0.992283940   dense_2_dense, ai_float,..</span></span>
<span id="cb24-8"><a href="#cb24-8" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   97.66%    6.643473148   5.626701355   0.992283940   dense_2_dense, ai_float,..</span></span>
<span id="cb24-9"><a href="#cb24-9" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          100.00%   0.000003622   0.000002494   0.000000541   dense_2_dense, ai_float,..</span></span>
<span id="cb24-10"><a href="#cb24-10" aria-hidden="true" tabindex="-1"></a><span class="op">---------------------------------------------------------------------------------------------------</span></span>
<span id="cb24-11"><a href="#cb24-11" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
<p>To avoid this situation, it is preferable as for a regressor model, to provide the “real” predicted values (as generated during the test of the trained model). In this case, the <code>ACC/CM</code> metrics are not significant.</p>
<div class="sourceCode" id="cb25"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb25-1"><a href="#cb25-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>float_model_wo_softmax<span class="op">&gt;</span> <span class="op">-</span>vi mnist_reduced_predicted_test_<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb25-2"><a href="#cb25-2" aria-hidden="true" tabindex="-1"></a><span class="op">..</span></span>
<span id="cb25-3"><a href="#cb25-3" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb25-4"><a href="#cb25-4" aria-hidden="true" tabindex="-1"></a><span class="op">---------------------------------------------------------------------------------------------------</span></span>
<span id="cb25-5"><a href="#cb25-5" aria-hidden="true" tabindex="-1"></a>Mode                acc       rmse          mae           l2r           tensor</span>
<span id="cb25-6"><a href="#cb25-6" aria-hidden="true" tabindex="-1"></a><span class="op">---------------------------------------------------------------------------------------------------</span></span>
<span id="cb25-7"><a href="#cb25-7" aria-hidden="true" tabindex="-1"></a>x86 C<span class="op">-</span>model <span class="co">#1      n.a.      0.000003622   0.000002494   0.000000541   dense_2_dense, ai_float,..</span></span>
<span id="cb25-8"><a href="#cb25-8" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   n.a.      0.000000000   0.000000000   0.000000000   dense_2_dense, ai_float,..</span></span>
<span id="cb25-9"><a href="#cb25-9" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          n.a.      0.000003622   0.000002494   0.000000541   dense_2_dense, ai_float,..</span></span>
<span id="cb25-10"><a href="#cb25-10" aria-hidden="true" tabindex="-1"></a><span class="op">---------------------------------------------------------------------------------------------------</span></span></code></pre></div>
</section>
</section>
<section id="quantized-model" class="level2">
<h2>Quantized model</h2>
<p>Previous floating point model has been quantized with a representative sub-set of the training data set. As the <code>tf.int8</code> option has been defined for the conversion of the inputs and outputs, the provided user data are automatically quantized before to feed the models. The predicted values are de-quantized before to compare them to the provided ground truth values, here one-hot encoded. As for the validation of the floating point model, the same comments about the provided metrics are applicable.</p>
<div class="sourceCode" id="cb26"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb26-1"><a href="#cb26-1" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> tflite_convert(keras_model, data):</span>
<span id="cb26-2"><a href="#cb26-2" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Quantize a Keras model (post-training quantization)&quot;&quot;&quot;</span></span>
<span id="cb26-3"><a href="#cb26-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb26-4"><a href="#cb26-4" aria-hidden="true" tabindex="-1"></a>  converter <span class="op">=</span> tf.lite.TFLiteConverter.from_keras_model(model)</span>
<span id="cb26-5"><a href="#cb26-5" aria-hidden="true" tabindex="-1"></a>  shape_in <span class="op">=</span> (<span class="dv">1</span>,) <span class="op">+</span> IN_SHAPE</span>
<span id="cb26-6"><a href="#cb26-6" aria-hidden="true" tabindex="-1"></a>        </span>
<span id="cb26-7"><a href="#cb26-7" aria-hidden="true" tabindex="-1"></a>  <span class="kw">def</span> rep_data_gen():</span>
<span id="cb26-8"><a href="#cb26-8" aria-hidden="true" tabindex="-1"></a>    <span class="cf">for</span> i <span class="kw">in</span> data[<span class="dv">0</span>:<span class="dv">100</span>]:</span>
<span id="cb26-9"><a href="#cb26-9" aria-hidden="true" tabindex="-1"></a>        f <span class="op">=</span> np.reshape(i, shape_in)</span>
<span id="cb26-10"><a href="#cb26-10" aria-hidden="true" tabindex="-1"></a>        tensor <span class="op">=</span> tf.convert_to_tensor(f, tf.float32)</span>
<span id="cb26-11"><a href="#cb26-11" aria-hidden="true" tabindex="-1"></a>        <span class="cf">yield</span> [tensor]</span>
<span id="cb26-12"><a href="#cb26-12" aria-hidden="true" tabindex="-1"></a>  </span>
<span id="cb26-13"><a href="#cb26-13" aria-hidden="true" tabindex="-1"></a>  converter.representative_dataset <span class="op">=</span> rep_data_gen   </span>
<span id="cb26-14"><a href="#cb26-14" aria-hidden="true" tabindex="-1"></a>  converter.optimizations <span class="op">=</span> [tf.lite.Optimize.DEFAULT]</span>
<span id="cb26-15"><a href="#cb26-15" aria-hidden="true" tabindex="-1"></a>  converter.target_spec.supported_ops <span class="op">=</span> [tf.lite.OpsSet.TFLITE_BUILTINS_INT8]</span>
<span id="cb26-16"><a href="#cb26-16" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb26-17"><a href="#cb26-17" aria-hidden="true" tabindex="-1"></a>  converter.inference_input_type <span class="op">=</span> np.int8</span>
<span id="cb26-18"><a href="#cb26-18" aria-hidden="true" tabindex="-1"></a>  converter.inference_output_type <span class="op">=</span> np.int8</span>
<span id="cb26-19"><a href="#cb26-19" aria-hidden="true" tabindex="-1"></a>  </span>
<span id="cb26-20"><a href="#cb26-20" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> converter.convert()</span></code></pre></div>
<div class="sourceCode" id="cb27"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb27-1"><a href="#cb27-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">&lt;</span>quantized_model<span class="op">&gt;</span> <span class="op">-</span>vi <span class="op">&lt;</span>data_directory<span class="op">&gt;/</span>mnist_reduced_test<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb27-2"><a href="#cb27-2" aria-hidden="true" tabindex="-1"></a>Neural Network Tools <span class="kw">for</span> STM32AI v1<span class="op">.</span><span class="fu">5</span><span class="op">.</span><span class="fu">1</span> <span class="op">(</span>STM<span class="op">.</span><span class="fu">ai</span> v7<span class="op">.</span><span class="fu">0</span><span class="op">.</span><span class="fu">0</span><span class="op">)</span></span>
<span id="cb27-3"><a href="#cb27-3" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-4"><a href="#cb27-4" aria-hidden="true" tabindex="-1"></a>Setting validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb27-5"><a href="#cb27-5" aria-hidden="true" tabindex="-1"></a> loading file<span class="op">:</span> <span class="op">&lt;</span>data_directory<span class="op">&gt;</span>\mnist_reduced_test<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb27-6"><a href="#cb27-6" aria-hidden="true" tabindex="-1"></a> <span class="op">-</span> samples are reshaped<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)</span> <span class="op">-&gt;</span> <span class="op">(</span>128<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)</span></span>
<span id="cb27-7"><a href="#cb27-7" aria-hidden="true" tabindex="-1"></a> <span class="op">-</span> samples are reshaped<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 10<span class="op">)</span> <span class="op">-&gt;</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)</span></span>
<span id="cb27-8"><a href="#cb27-8" aria-hidden="true" tabindex="-1"></a> I<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>128<span class="op">,</span> 28<span class="op">,</span> 28<span class="op">,</span> 1<span class="op">)/</span>int8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[-</span>128<span class="op">,</span> 127<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[-</span>92<span class="op">.</span><span class="fu">544</span><span class="op">,</span> 81<span class="op">.</span><span class="fu">165</span><span class="op">],</span></span>
<span id="cb27-9"><a href="#cb27-9" aria-hidden="true" tabindex="-1"></a>       scale<span class="op">=</span>0<span class="op">.</span><span class="fu">00392157</span> zp<span class="op">=-</span>128<span class="op">,</span> input_1</span>
<span id="cb27-10"><a href="#cb27-10" aria-hidden="true" tabindex="-1"></a> O<span class="op">[</span>1<span class="op">]:</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>float32<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[</span>0<span class="op">.</span><span class="fu">000</span><span class="op">,</span> 1<span class="op">.</span><span class="fu">000</span><span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[</span>0<span class="op">.</span><span class="fu">100</span><span class="op">,</span> 0<span class="op">.</span><span class="fu">300</span><span class="op">],</span></span>
<span id="cb27-11"><a href="#cb27-11" aria-hidden="true" tabindex="-1"></a>       nl_4_fmt_conv</span>
<span id="cb27-12"><a href="#cb27-12" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-13"><a href="#cb27-13" aria-hidden="true" tabindex="-1"></a>Running the STM AI c<span class="op">-</span>model <span class="op">(</span>AI RUNNER<span class="op">)...(</span>name<span class="op">=</span>network<span class="op">,</span> mode<span class="op">=</span>x86<span class="op">)</span></span>
<span id="cb27-14"><a href="#cb27-14" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb27-15"><a href="#cb27-15" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-16"><a href="#cb27-16" aria-hidden="true" tabindex="-1"></a>Running the TFlite model<span class="op">...</span></span>
<span id="cb27-17"><a href="#cb27-17" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-18"><a href="#cb27-18" aria-hidden="true" tabindex="-1"></a>Saving validation <span class="kw">data</span><span class="op">...</span></span>
<span id="cb27-19"><a href="#cb27-19" aria-hidden="true" tabindex="-1"></a> output directory<span class="op">:</span> <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span></span>
<span id="cb27-20"><a href="#cb27-20" aria-hidden="true" tabindex="-1"></a> creating <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_val_io<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb27-21"><a href="#cb27-21" aria-hidden="true" tabindex="-1"></a> m_outputs_1<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>int8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[-</span>128<span class="op">,</span> 127<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[-</span>102<span class="op">.</span><span class="fu">458</span><span class="op">,</span> 73<span class="op">.</span><span class="fu">581</span><span class="op">],</span></span>
<span id="cb27-22"><a href="#cb27-22" aria-hidden="true" tabindex="-1"></a>              nl_4_fmt_conv</span>
<span id="cb27-23"><a href="#cb27-23" aria-hidden="true" tabindex="-1"></a> c_outputs_1<span class="op">:</span> <span class="op">(</span>128<span class="op">,</span> 1<span class="op">,</span> 1<span class="op">,</span> 10<span class="op">)/</span>int8<span class="op">,</span> min<span class="op">/</span>max<span class="op">=[-</span>128<span class="op">,</span> 127<span class="op">],</span> mean<span class="op">/</span>std<span class="op">=[-</span>102<span class="op">.</span><span class="fu">458</span><span class="op">,</span> 73<span class="op">.</span><span class="fu">581</span><span class="op">],</span></span>
<span id="cb27-24"><a href="#cb27-24" aria-hidden="true" tabindex="-1"></a>              scale<span class="op">=</span>0<span class="op">.</span><span class="fu">00390625</span> zp<span class="op">=-</span>128<span class="op">,</span> nl_4_fmt_conv</span>
<span id="cb27-25"><a href="#cb27-25" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-26"><a href="#cb27-26" aria-hidden="true" tabindex="-1"></a>Computing the metrics<span class="op">...</span></span>
<span id="cb27-27"><a href="#cb27-27" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-28"><a href="#cb27-28" aria-hidden="true" tabindex="-1"></a> Accuracy report <span class="co">#1 for the generated x86 C-model</span></span>
<span id="cb27-29"><a href="#cb27-29" aria-hidden="true" tabindex="-1"></a> <span class="op">------------------------------------------------------------------------------------</span></span>
<span id="cb27-30"><a href="#cb27-30" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> <span class="kw">data</span> <span class="fu">type</span> is different<span class="op">:</span> <span class="fu">r</span><span class="op">/</span>float32 instead p<span class="op">/</span>int8</span>
<span id="cb27-31"><a href="#cb27-31" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> p<span class="op">/</span>int8 <span class="kw">data</span> are dequantized with s<span class="op">=</span>0<span class="op">.</span><span class="fu">003906</span> zp<span class="op">=-</span>128</span>
<span id="cb27-32"><a href="#cb27-32" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> computed against the provided ground truth values</span>
<span id="cb27-33"><a href="#cb27-33" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 128 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb27-34"><a href="#cb27-34" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-35"><a href="#cb27-35" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>97<span class="op">.</span><span class="fu">66</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">054981638</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">010229493</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">180712447</span></span>
<span id="cb27-36"><a href="#cb27-36" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-37"><a href="#cb27-37" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>128 samples<span class="op">)</span></span>
<span id="cb27-38"><a href="#cb27-38" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb27-39"><a href="#cb27-39" aria-hidden="true" tabindex="-1"></a>  C0       11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-40"><a href="#cb27-40" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>   19    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-41"><a href="#cb27-41" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-42"><a href="#cb27-42" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-43"><a href="#cb27-43" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   15    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-44"><a href="#cb27-44" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-45"><a href="#cb27-45" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-46"><a href="#cb27-46" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-47"><a href="#cb27-47" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   17    <span class="op">.</span></span>
<span id="cb27-48"><a href="#cb27-48" aria-hidden="true" tabindex="-1"></a>  C9        1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10</span>
<span id="cb27-49"><a href="#cb27-49" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-50"><a href="#cb27-50" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-51"><a href="#cb27-51" aria-hidden="true" tabindex="-1"></a> Accuracy report <span class="co">#1 for the reference model</span></span>
<span id="cb27-52"><a href="#cb27-52" aria-hidden="true" tabindex="-1"></a> <span class="op">------------------------------------------------------------------------------------</span></span>
<span id="cb27-53"><a href="#cb27-53" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> <span class="kw">data</span> <span class="fu">type</span> is different<span class="op">:</span> <span class="fu">r</span><span class="op">/</span>float32 instead p<span class="op">/</span>int8</span>
<span id="cb27-54"><a href="#cb27-54" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> p<span class="op">/</span>int8 <span class="kw">data</span> are dequantized with s<span class="op">=</span>0<span class="op">.</span><span class="fu">003906</span> zp<span class="op">=-</span>128</span>
<span id="cb27-55"><a href="#cb27-55" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> computed against the provided ground truth values</span>
<span id="cb27-56"><a href="#cb27-56" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 128 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb27-57"><a href="#cb27-57" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-58"><a href="#cb27-58" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>97<span class="op">.</span><span class="fu">66</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">054981638</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">010229493</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">180712447</span></span>
<span id="cb27-59"><a href="#cb27-59" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-60"><a href="#cb27-60" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>128 samples<span class="op">)</span></span>
<span id="cb27-61"><a href="#cb27-61" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb27-62"><a href="#cb27-62" aria-hidden="true" tabindex="-1"></a>  C0       11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-63"><a href="#cb27-63" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>   19    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-64"><a href="#cb27-64" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-65"><a href="#cb27-65" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   11    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-66"><a href="#cb27-66" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   15    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-67"><a href="#cb27-67" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-68"><a href="#cb27-68" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-69"><a href="#cb27-69" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-70"><a href="#cb27-70" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   17    <span class="op">.</span></span>
<span id="cb27-71"><a href="#cb27-71" aria-hidden="true" tabindex="-1"></a>  C9        1    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10</span>
<span id="cb27-72"><a href="#cb27-72" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-73"><a href="#cb27-73" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-74"><a href="#cb27-74" aria-hidden="true" tabindex="-1"></a> Cross accuracy report <span class="co">#1 (reference vs C-model)</span></span>
<span id="cb27-75"><a href="#cb27-75" aria-hidden="true" tabindex="-1"></a> <span class="op">-------------------------------------------------------------------------------------</span></span>
<span id="cb27-76"><a href="#cb27-76" aria-hidden="true" tabindex="-1"></a> notes<span class="op">:</span> <span class="op">-</span> <span class="fu">r</span><span class="op">/</span>int8 <span class="kw">data</span> are dequantized with s<span class="op">=</span>0<span class="op">.</span><span class="fu">003906</span> zp<span class="op">=-</span>128</span>
<span id="cb27-77"><a href="#cb27-77" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> p<span class="op">/</span>int8 <span class="kw">data</span> are dequantized with s<span class="op">=</span>0<span class="op">.</span><span class="fu">003906</span> zp<span class="op">=-</span>128</span>
<span id="cb27-78"><a href="#cb27-78" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> the output of the reference model is used as ground truth<span class="op">/</span>reference value</span>
<span id="cb27-79"><a href="#cb27-79" aria-hidden="true" tabindex="-1"></a>        <span class="op">-</span> 128 samples <span class="op">(</span>10 items per sample<span class="op">)</span></span>
<span id="cb27-80"><a href="#cb27-80" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-81"><a href="#cb27-81" aria-hidden="true" tabindex="-1"></a>  acc<span class="op">=</span>100<span class="op">.</span><span class="fu">00</span><span class="op">%,</span> rmse<span class="op">=</span>0<span class="op">.</span><span class="fu">000000000</span><span class="op">,</span> mae<span class="op">=</span>0<span class="op">.</span><span class="fu">000000000</span><span class="op">,</span> l2r<span class="op">=</span>0<span class="op">.</span><span class="fu">000000000</span></span>
<span id="cb27-82"><a href="#cb27-82" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-83"><a href="#cb27-83" aria-hidden="true" tabindex="-1"></a>  10 classes <span class="op">(</span>128 samples<span class="op">)</span></span>
<span id="cb27-84"><a href="#cb27-84" aria-hidden="true" tabindex="-1"></a>  <span class="op">----------------------------------------------------------</span></span>
<span id="cb27-85"><a href="#cb27-85" aria-hidden="true" tabindex="-1"></a>  C0       12    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-86"><a href="#cb27-86" aria-hidden="true" tabindex="-1"></a>  C1        <span class="op">.</span>   19    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-87"><a href="#cb27-87" aria-hidden="true" tabindex="-1"></a>  C2        <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-88"><a href="#cb27-88" aria-hidden="true" tabindex="-1"></a>  C3        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   12    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-89"><a href="#cb27-89" aria-hidden="true" tabindex="-1"></a>  C4        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   16    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-90"><a href="#cb27-90" aria-hidden="true" tabindex="-1"></a>  C5        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    7    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-91"><a href="#cb27-91" aria-hidden="true" tabindex="-1"></a>  C6        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-92"><a href="#cb27-92" aria-hidden="true" tabindex="-1"></a>  C7        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    9    <span class="op">.</span>    <span class="op">.</span></span>
<span id="cb27-93"><a href="#cb27-93" aria-hidden="true" tabindex="-1"></a>  C8        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   17    <span class="op">.</span></span>
<span id="cb27-94"><a href="#cb27-94" aria-hidden="true" tabindex="-1"></a>  C9        <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>    <span class="op">.</span>   10</span>
<span id="cb27-95"><a href="#cb27-95" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-96"><a href="#cb27-96" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-97"><a href="#cb27-97" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb27-98"><a href="#cb27-98" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">------------------------</span></span>
<span id="cb27-99"><a href="#cb27-99" aria-hidden="true" tabindex="-1"></a>Output              acc       rmse             l2r            tensor</span>
<span id="cb27-100"><a href="#cb27-100" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">------------------------</span></span>
<span id="cb27-101"><a href="#cb27-101" aria-hidden="true" tabindex="-1"></a>x86 c<span class="op">-</span>model <span class="co">#1      97.66%    0.054981638 ... 0.180712447 ... nl_4_fmt_conv, ai_i8,...</span></span>
<span id="cb27-102"><a href="#cb27-102" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   97.66%    0.054981638 ... 0.180712447 ... nl_4_fmt_conv, ai_i8,...</span></span>
<span id="cb27-103"><a href="#cb27-103" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          100.00%   0.000000000 ... 0.000000000 ... nl_4_fmt_conv, ai_i8,...</span></span>
<span id="cb27-104"><a href="#cb27-104" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------</span> <span class="op">...</span> <span class="op">-----------</span> <span class="op">...</span> <span class="op">------------------------</span></span>
<span id="cb27-105"><a href="#cb27-105" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb27-106"><a href="#cb27-106" aria-hidden="true" tabindex="-1"></a>Creating report file <span class="op">&lt;</span>output<span class="op">-</span>directory<span class="op">-</span>path<span class="op">&gt;</span>\network_validate_report<span class="op">.</span><span class="fu">txt</span></span></code></pre></div>
</section>
<section id="quantized-keras-model" class="level2">
<h2>Quantized keras model</h2>
<p>Validation of the <a href="quantization.html#ref_quantize_cmd">Quantized Keras model [QUANT]</a> is a particular case. The particularity is that the reshaped Keras model is <em>always</em> executed as a floating-point model and compared to the quantized c-model. Only float32 input data can be provided in this case.</p>
<div id="fig:id_val_st_q_keras" class="fignos">
<figure>
<img src="" property="center" style="width:85.0%" alt="Figure 8: Computation of the metric (Quantized Keras model)" /><figcaption aria-hidden="true"><span>Figure 8:</span> Computation of the metric (Quantized Keras model)</figcaption>
</figure>
</div>
<p>With the reference data generated by the quantization process, the errors are close to zero for the C-model, because the outputs have been generated by a Keras model with the fake quantized layers. The imported reshaped Keras model is a model w/o these layers. It is strictly equivalent to the original floating-point Keras model.</p>
<div class="sourceCode" id="cb28"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb28-1"><a href="#cb28-1" aria-hidden="true" tabindex="-1"></a>$ stm32 validate <span class="op">&lt;</span>reshaped_keras_model<span class="op">.</span><span class="fu">h5</span><span class="op">&gt;</span> <span class="op">-</span>q <span class="op">&lt;</span>quant_conf_Q<span class="op">.</span><span class="fu">json</span><span class="op">&gt;</span> <span class="op">-</span>vi references<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb28-2"><a href="#cb28-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb28-3"><a href="#cb28-3" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb28-4"><a href="#cb28-4" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------------------------------------------------------------</span></span>
<span id="cb28-5"><a href="#cb28-5" aria-hidden="true" tabindex="-1"></a>Mode                acc       rmse          mae           l2r           tensor</span>
<span id="cb28-6"><a href="#cb28-6" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------------------------------------------------------------</span></span>
<span id="cb28-7"><a href="#cb28-7" aria-hidden="true" tabindex="-1"></a>x86 C<span class="op">-</span>model <span class="co">#1      100.00%   0.000000002   0.000000000   0.000000007   softmax_8, ai_float, ..</span></span>
<span id="cb28-8"><a href="#cb28-8" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   100.00%   0.001213685   0.000121268   0.003879170   softmax_8, ai_float, ..</span></span>
<span id="cb28-9"><a href="#cb28-9" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          100.00%   0.001213685   0.000121268   0.003878688   softmax_8, ai_float, ..</span></span>
<span id="cb28-10"><a href="#cb28-10" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------------------------------------------------------------</span></span></code></pre></div>
<div class="sourceCode" id="cb29"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb29-1"><a href="#cb29-1" aria-hidden="true" tabindex="-1"></a>$ stm32 validate <span class="op">&lt;</span>reshaped_keras_model<span class="op">.</span><span class="fu">h5</span><span class="op">&gt;</span> <span class="op">-</span>q <span class="op">&lt;</span>quant_conf_Q<span class="op">.</span><span class="fu">json</span><span class="op">&gt;</span> <span class="op">-</span>vi mnist_reduced_test<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb29-2"><a href="#cb29-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb29-3"><a href="#cb29-3" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb29-4"><a href="#cb29-4" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------------------------------------------------------------</span></span>
<span id="cb29-5"><a href="#cb29-5" aria-hidden="true" tabindex="-1"></a>Mode                acc       rmse          mae           l2r           tensor</span>
<span id="cb29-6"><a href="#cb29-6" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------------------------------------------------------------</span></span>
<span id="cb29-7"><a href="#cb29-7" aria-hidden="true" tabindex="-1"></a>x86 C<span class="op">-</span>model <span class="co">#1      99.22%    0.038811572   0.002098498   0.123097971   softmax_8, ai_float, ..</span></span>
<span id="cb29-8"><a href="#cb29-8" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   99.22%    0.038617868   0.002059023   0.122468531   softmax_8, ai_float, ..</span></span>
<span id="cb29-9"><a href="#cb29-9" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          100.00%   0.000692510   0.000046946   0.002196421   softmax_8, ai_float, ..</span></span>
<span id="cb29-10"><a href="#cb29-10" aria-hidden="true" tabindex="-1"></a><span class="op">-----------------------------------------------------------------------------------------------</span></span>
<span id="cb29-11"><a href="#cb29-11" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span></code></pre></div>
</section>
<section id="model-with-multiple-io" class="level2">
<h2>Model with multiple IO</h2>
<p>No specific process is applied, all metrics are calculated independently for each outputs.</p>
<div class="sourceCode" id="cb30"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb30-1"><a href="#cb30-1" aria-hidden="true" tabindex="-1"></a>$ stm32ai validate <span class="op">-</span>m <span class="op">&lt;</span>model_with_2_outputs<span class="op">&gt;</span> <span class="op">-</span>vi test_multiple_io<span class="op">.</span><span class="fu">npz</span></span>
<span id="cb30-2"><a href="#cb30-2" aria-hidden="true" tabindex="-1"></a><span class="op">...</span></span>
<span id="cb30-3"><a href="#cb30-3" aria-hidden="true" tabindex="-1"></a>Evaluation report <span class="op">(</span>summary<span class="op">)</span></span>
<span id="cb30-4"><a href="#cb30-4" aria-hidden="true" tabindex="-1"></a><span class="op">-------------------------------------------------------------------------------------------------</span></span>
<span id="cb30-5"><a href="#cb30-5" aria-hidden="true" tabindex="-1"></a>Mode                acc    rmse          mae           l2r           tensor</span>
<span id="cb30-6"><a href="#cb30-6" aria-hidden="true" tabindex="-1"></a><span class="op">-------------------------------------------------------------------------------------------------</span></span>
<span id="cb30-7"><a href="#cb30-7" aria-hidden="true" tabindex="-1"></a>x86 C<span class="op">-</span>model <span class="co">#1      n.a.   0.000000000   0.000000000   0.000000000   add_1, ai_float, ..</span></span>
<span id="cb30-8"><a href="#cb30-8" aria-hidden="true" tabindex="-1"></a>x86 C<span class="op">-</span>model <span class="co">#2      n.a.   0.000000000   0.000000000   0.000000000   multiply_1, ai_float, ..</span></span>
<span id="cb30-9"><a href="#cb30-9" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#1   n.a.   0.000000000   0.000000000   0.000000000   add_1, ai_float, ..</span></span>
<span id="cb30-10"><a href="#cb30-10" aria-hidden="true" tabindex="-1"></a>original model <span class="co">#2   n.a.   0.000000000   0.000000000   0.000000000   multiply_1, ai_float, ..</span></span>
<span id="cb30-11"><a href="#cb30-11" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#1          n.a.   0.000000000   0.000000000   0.000000000   add_1, ai_float, ..</span></span>
<span id="cb30-12"><a href="#cb30-12" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="co">#2          n.a.   0.000000000   0.000000000   0.000000000   multiply_1, ai_float, ..</span></span>
<span id="cb30-13"><a href="#cb30-13" aria-hidden="true" tabindex="-1"></a><span class="op">-------------------------------------------------------------------------------------------------</span></span>
<span id="cb30-14"><a href="#cb30-14" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb30-15"><a href="#cb30-15" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross <span class="op">(</span>l2r<span class="op">)</span> <span class="co">#1 error : 0.00000000e+00 (expected to be &lt; 0.01)</span></span></code></pre></div>
</section>
</section>
<section id="ref_script_ex" class="level1">
<h1>Post-processing example</h1>
<p>As illustrated below, this section provides a typical custom Python script example to read the generated data and to build new element-wise metrics: <code>variance</code>, <code>f1_score</code>,.. (thanks to the <em>numpy</em> and <em>sklearn.metrics</em> Python modules). <code>acc</code>, <code>rmse</code>, <code>mae</code> and <code>l2r</code> are also provided to illustrate how these metrics are computed.</p>
<div class="sourceCode" id="cb31"><pre class="sourceCode powershell"><code class="sourceCode powershell"><span id="cb31-1"><a href="#cb31-1" aria-hidden="true" tabindex="-1"></a>$ python custom_metrics<span class="op">.</span><span class="fu">py</span></span>
<span id="cb31-2"><a href="#cb31-2" aria-hidden="true" tabindex="-1"></a>Read reference NPZ file <span class="st">&quot;mnist_test.npz&quot;</span><span class="op">...</span></span>
<span id="cb31-3"><a href="#cb31-3" aria-hidden="true" tabindex="-1"></a>Read generated NPZ file <span class="st">&quot;./stm32ai_output\network_val_io.npz&quot;</span><span class="op">...</span></span>
<span id="cb31-4"><a href="#cb31-4" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb31-5"><a href="#cb31-5" aria-hidden="true" tabindex="-1"></a>Evaluation report</span>
<span id="cb31-6"><a href="#cb31-6" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb31-7"><a href="#cb31-7" aria-hidden="true" tabindex="-1"></a>               acc       rmse      mae       <span class="kw">var</span>       f1_score  l2r</span>
<span id="cb31-8"><a href="#cb31-8" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb31-9"><a href="#cb31-9" aria-hidden="true" tabindex="-1"></a>C<span class="op">-</span>model        98<span class="op">.</span><span class="fu">4</span><span class="op">%</span>     0<span class="op">.</span><span class="fu">064472</span>  0<span class="op">.</span><span class="fu">013081</span>  0<span class="op">.</span><span class="fu">004160</span>  0<span class="op">.</span><span class="fu">984033</span>  0<span class="op">.</span><span class="fu">21363260</span></span>
<span id="cb31-10"><a href="#cb31-10" aria-hidden="true" tabindex="-1"></a>original model 98<span class="op">.</span><span class="fu">4</span><span class="op">%</span>     0<span class="op">.</span><span class="fu">064472</span>  0<span class="op">.</span><span class="fu">013081</span>  0<span class="op">.</span><span class="fu">004160</span>  0<span class="op">.</span><span class="fu">984033</span>  0<span class="op">.</span><span class="fu">21363260</span></span>
<span id="cb31-11"><a href="#cb31-11" aria-hidden="true" tabindex="-1"></a>X<span class="op">-</span>cross        100<span class="op">.</span><span class="fu">0</span><span class="op">%</span>    0<span class="op">.</span><span class="fu">000000</span>  0<span class="op">.</span><span class="fu">000000</span>  0<span class="op">.</span><span class="fu">000000</span>  1<span class="op">.</span><span class="fu">000000</span>  0<span class="op">.</span><span class="fu">00000000</span></span>
<span id="cb31-12"><a href="#cb31-12" aria-hidden="true" tabindex="-1"></a><span class="op">--------------------------------------------------------------------------------</span></span>
<span id="cb31-13"><a href="#cb31-13" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb31-14"><a href="#cb31-14" aria-hidden="true" tabindex="-1"></a>              precision    recall  f1<span class="op">-</span>score   support</span>
<span id="cb31-15"><a href="#cb31-15" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb31-16"><a href="#cb31-16" aria-hidden="true" tabindex="-1"></a>          c0       0<span class="op">.</span><span class="fu">92</span>      1<span class="op">.</span><span class="fu">00</span>      0<span class="op">.</span><span class="fu">96</span>        12</span>
<span id="cb31-17"><a href="#cb31-17" aria-hidden="true" tabindex="-1"></a>          c1       1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>        19</span>
<span id="cb31-18"><a href="#cb31-18" aria-hidden="true" tabindex="-1"></a>          c2       1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>        16</span>
<span id="cb31-19"><a href="#cb31-19" aria-hidden="true" tabindex="-1"></a>          c3       0<span class="op">.</span><span class="fu">92</span>      1<span class="op">.</span><span class="fu">00</span>      0<span class="op">.</span><span class="fu">96</span>        11</span>
<span id="cb31-20"><a href="#cb31-20" aria-hidden="true" tabindex="-1"></a>          c4       1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>        15</span>
<span id="cb31-21"><a href="#cb31-21" aria-hidden="true" tabindex="-1"></a>          c5       1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>         7</span>
<span id="cb31-22"><a href="#cb31-22" aria-hidden="true" tabindex="-1"></a>          c6       1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>        10</span>
<span id="cb31-23"><a href="#cb31-23" aria-hidden="true" tabindex="-1"></a>          c7       1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>      1<span class="op">.</span><span class="fu">00</span>         9</span>
<span id="cb31-24"><a href="#cb31-24" aria-hidden="true" tabindex="-1"></a>          c8       1<span class="op">.</span><span class="fu">00</span>      0<span class="op">.</span><span class="fu">94</span>      0<span class="op">.</span><span class="fu">97</span>        18</span>
<span id="cb31-25"><a href="#cb31-25" aria-hidden="true" tabindex="-1"></a>          c9       1<span class="op">.</span><span class="fu">00</span>      0<span class="op">.</span><span class="fu">91</span>      0<span class="op">.</span><span class="fu">95</span>        11</span>
<span id="cb31-26"><a href="#cb31-26" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb31-27"><a href="#cb31-27" aria-hidden="true" tabindex="-1"></a>    accuracy                           0<span class="op">.</span><span class="fu">98</span>       128</span>
<span id="cb31-28"><a href="#cb31-28" aria-hidden="true" tabindex="-1"></a>   macro avg       0<span class="op">.</span><span class="fu">98</span>      0<span class="op">.</span><span class="fu">99</span>      0<span class="op">.</span><span class="fu">98</span>       128</span>
<span id="cb31-29"><a href="#cb31-29" aria-hidden="true" tabindex="-1"></a>weighted avg       0<span class="op">.</span><span class="fu">99</span>      0<span class="op">.</span><span class="fu">98</span>      0<span class="op">.</span><span class="fu">98</span>       128</span></code></pre></div>
<p>Full code of the custom Python script.</p>
<div class="sourceCode" id="cb32"><pre class="sourceCode python"><code class="sourceCode python"><span id="cb32-1"><a href="#cb32-1" aria-hidden="true" tabindex="-1"></a><span class="co"># -*- coding: utf-8 -*-</span></span>
<span id="cb32-2"><a href="#cb32-2" aria-hidden="true" tabindex="-1"></a><span class="co">&quot;&quot;&quot;</span></span>
<span id="cb32-3"><a href="#cb32-3" aria-hidden="true" tabindex="-1"></a><span class="co">Implement custom metrics</span></span>
<span id="cb32-4"><a href="#cb32-4" aria-hidden="true" tabindex="-1"></a><span class="co">&quot;&quot;&quot;</span></span>
<span id="cb32-5"><a href="#cb32-5" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> __future__ <span class="im">import</span> absolute_import</span>
<span id="cb32-6"><a href="#cb32-6" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> __future__ <span class="im">import</span> division</span>
<span id="cb32-7"><a href="#cb32-7" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> __future__ <span class="im">import</span> print_function</span>
<span id="cb32-8"><a href="#cb32-8" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-9"><a href="#cb32-9" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> os</span>
<span id="cb32-10"><a href="#cb32-10" aria-hidden="true" tabindex="-1"></a><span class="im">import</span> numpy <span class="im">as</span> np</span>
<span id="cb32-11"><a href="#cb32-11" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-12"><a href="#cb32-12" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> mean_squared_error</span>
<span id="cb32-13"><a href="#cb32-13" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> accuracy_score</span>
<span id="cb32-14"><a href="#cb32-14" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> f1_score</span>
<span id="cb32-15"><a href="#cb32-15" aria-hidden="true" tabindex="-1"></a><span class="im">from</span> sklearn.metrics <span class="im">import</span> classification_report</span>
<span id="cb32-16"><a href="#cb32-16" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-17"><a href="#cb32-17" aria-hidden="true" tabindex="-1"></a>OUTPUT_DIR <span class="op">=</span> <span class="st">&#39;./stm32ai_output&#39;</span></span>
<span id="cb32-18"><a href="#cb32-18" aria-hidden="true" tabindex="-1"></a>NETWORK_NAME <span class="op">=</span> <span class="st">&#39;network&#39;</span></span>
<span id="cb32-19"><a href="#cb32-19" aria-hidden="true" tabindex="-1"></a>REFERENCE_NPZ <span class="op">=</span> <span class="st">&#39;mnist_test.npz&#39;</span></span>
<span id="cb32-20"><a href="#cb32-20" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-21"><a href="#cb32-21" aria-hidden="true" tabindex="-1"></a><span class="co"># metrics</span></span>
<span id="cb32-22"><a href="#cb32-22" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-23"><a href="#cb32-23" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> mse(ref, pred):</span>
<span id="cb32-24"><a href="#cb32-24" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return Mean Squared Error (MSE).&quot;&quot;&quot;</span></span>
<span id="cb32-25"><a href="#cb32-25" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> ((ref <span class="op">-</span> pred).astype(np.float64) <span class="op">**</span> <span class="dv">2</span>).mean()</span>
<span id="cb32-26"><a href="#cb32-26" aria-hidden="true" tabindex="-1"></a>  </span>
<span id="cb32-27"><a href="#cb32-27" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> rmse(ref, pred):</span>
<span id="cb32-28"><a href="#cb32-28" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return Root Mean Squared Error (RMSE).&quot;&quot;&quot;</span></span>
<span id="cb32-29"><a href="#cb32-29" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> np.sqrt(((ref <span class="op">-</span> pred).astype(np.float64) <span class="op">**</span> <span class="dv">2</span>).mean())</span>
<span id="cb32-30"><a href="#cb32-30" aria-hidden="true" tabindex="-1"></a>  </span>
<span id="cb32-31"><a href="#cb32-31" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> mae(ref, pred):</span>
<span id="cb32-32"><a href="#cb32-32" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return Mean Absolute Error (MAE).&quot;&quot;&quot;</span></span>
<span id="cb32-33"><a href="#cb32-33" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> (np.<span class="bu">abs</span>(ref <span class="op">-</span> pred).astype(np.float64)).mean()</span>
<span id="cb32-34"><a href="#cb32-34" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-35"><a href="#cb32-35" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> var(ref, pred):</span>
<span id="cb32-36"><a href="#cb32-36" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Return Variance&quot;&quot;&quot;</span></span>
<span id="cb32-37"><a href="#cb32-37" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> np.var((ref <span class="op">-</span> pred), dtype<span class="op">=</span>np.float64, ddof<span class="op">=</span><span class="dv">1</span>)</span>
<span id="cb32-38"><a href="#cb32-38" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-39"><a href="#cb32-39" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> acc(ref, pred):</span>
<span id="cb32-40"><a href="#cb32-40" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Classification accuracy (ACC).&quot;&quot;&quot;</span></span>
<span id="cb32-41"><a href="#cb32-41" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> accuracy_score(np.argmax(ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(pred, axis<span class="op">=</span><span class="dv">1</span>))</span>
<span id="cb32-42"><a href="#cb32-42" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-43"><a href="#cb32-43" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> f1_s(ref, pred, average<span class="op">=</span><span class="st">&#39;macro&#39;</span>):</span>
<span id="cb32-44"><a href="#cb32-44" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Compute the F1 score, also known as balanced F-score or F-measure (F1)&quot;&quot;&quot;</span></span>
<span id="cb32-45"><a href="#cb32-45" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> f1_score(np.argmax(ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(pred, axis<span class="op">=</span><span class="dv">1</span>), average<span class="op">=</span>average)</span>
<span id="cb32-46"><a href="#cb32-46" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-47"><a href="#cb32-47" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> l2r(ref, pred):</span>
<span id="cb32-48"><a href="#cb32-48" aria-hidden="true" tabindex="-1"></a>  <span class="co">&quot;&quot;&quot;Compute L2 relative error&quot;&quot;&quot;</span></span>
<span id="cb32-49"><a href="#cb32-49" aria-hidden="true" tabindex="-1"></a>  <span class="kw">def</span> magnitude(v):</span>
<span id="cb32-50"><a href="#cb32-50" aria-hidden="true" tabindex="-1"></a>    <span class="cf">return</span> np.sqrt(np.<span class="bu">sum</span>(np.square(v).flatten()))</span>
<span id="cb32-51"><a href="#cb32-51" aria-hidden="true" tabindex="-1"></a>  mag <span class="op">=</span> magnitude(pred) <span class="op">+</span> np.finfo(np.float32).eps</span>
<span id="cb32-52"><a href="#cb32-52" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> magnitude(ref <span class="op">-</span> pred) <span class="op">/</span> mag</span>
<span id="cb32-53"><a href="#cb32-53" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-54"><a href="#cb32-54" aria-hidden="true" tabindex="-1"></a>  </span>
<span id="cb32-55"><a href="#cb32-55" aria-hidden="true" tabindex="-1"></a><span class="co"># Read reference values</span></span>
<span id="cb32-56"><a href="#cb32-56" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-57"><a href="#cb32-57" aria-hidden="true" tabindex="-1"></a>fname <span class="op">=</span> REFERENCE_NPZ</span>
<span id="cb32-58"><a href="#cb32-58" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;Read reference NPZ file &quot;</span><span class="sc">{}</span><span class="st">&quot;...&#39;</span>.<span class="bu">format</span>(fname))</span>
<span id="cb32-59"><a href="#cb32-59" aria-hidden="true" tabindex="-1"></a>arrays <span class="op">=</span> np.load(fname)</span>
<span id="cb32-60"><a href="#cb32-60" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-61"><a href="#cb32-61" aria-hidden="true" tabindex="-1"></a>i_ref <span class="op">=</span> arrays[<span class="st">&#39;x_test&#39;</span>]</span>
<span id="cb32-62"><a href="#cb32-62" aria-hidden="true" tabindex="-1"></a>r_ref <span class="op">=</span> arrays[<span class="st">&#39;y_test&#39;</span>]</span>
<span id="cb32-63"><a href="#cb32-63" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-64"><a href="#cb32-64" aria-hidden="true" tabindex="-1"></a><span class="co"># Read the generated inputs and predicted samples (origninal &amp; C models)</span></span>
<span id="cb32-65"><a href="#cb32-65" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-66"><a href="#cb32-66" aria-hidden="true" tabindex="-1"></a>fname <span class="op">=</span> os.path.join(OUTPUT_DIR, NETWORK_NAME <span class="op">+</span> <span class="st">&#39;_val_io.npz&#39;</span>)</span>
<span id="cb32-67"><a href="#cb32-67" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;Read generated NPZ file &quot;</span><span class="sc">{}</span><span class="st">&quot;...&#39;</span>.<span class="bu">format</span>(fname))</span>
<span id="cb32-68"><a href="#cb32-68" aria-hidden="true" tabindex="-1"></a>arrays <span class="op">=</span> np.load(fname)</span>
<span id="cb32-69"><a href="#cb32-69" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-70"><a href="#cb32-70" aria-hidden="true" tabindex="-1"></a>i_  <span class="op">=</span> arrays[<span class="st">&#39;m_inputs_1&#39;</span>]</span>
<span id="cb32-71"><a href="#cb32-71" aria-hidden="true" tabindex="-1"></a>ic_  <span class="op">=</span> arrays[<span class="st">&#39;c_inputs_1&#39;</span>]</span>
<span id="cb32-72"><a href="#cb32-72" aria-hidden="true" tabindex="-1"></a>p_  <span class="op">=</span> arrays[<span class="st">&#39;c_outputs_1&#39;</span>]</span>
<span id="cb32-73"><a href="#cb32-73" aria-hidden="true" tabindex="-1"></a>pm_ <span class="op">=</span> arrays[<span class="st">&#39;m_outputs_1&#39;</span>]</span>
<span id="cb32-74"><a href="#cb32-74" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-75"><a href="#cb32-75" aria-hidden="true" tabindex="-1"></a><span class="co"># calculate metrics</span></span>
<span id="cb32-76"><a href="#cb32-76" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-77"><a href="#cb32-77" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> build_metrics(ref, pred):</span>
<span id="cb32-78"><a href="#cb32-78" aria-hidden="true" tabindex="-1"></a>  res <span class="op">=</span> {}</span>
<span id="cb32-79"><a href="#cb32-79" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;acc&#39;</span>] <span class="op">=</span> acc(ref, pred)</span>
<span id="cb32-80"><a href="#cb32-80" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;var&#39;</span>] <span class="op">=</span> var(ref, pred)</span>
<span id="cb32-81"><a href="#cb32-81" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;f1_score&#39;</span>] <span class="op">=</span> f1_s(ref, pred)</span>
<span id="cb32-82"><a href="#cb32-82" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;rmse&#39;</span>] <span class="op">=</span> rmse(ref, pred)</span>
<span id="cb32-83"><a href="#cb32-83" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;mae&#39;</span>] <span class="op">=</span> mae(ref, pred)</span>
<span id="cb32-84"><a href="#cb32-84" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;mse&#39;</span>] <span class="op">=</span> mse(ref, pred)</span>
<span id="cb32-85"><a href="#cb32-85" aria-hidden="true" tabindex="-1"></a>  res[<span class="st">&#39;l2r&#39;</span>] <span class="op">=</span> l2r(ref, pred)</span>
<span id="cb32-86"><a href="#cb32-86" aria-hidden="true" tabindex="-1"></a>  <span class="cf">return</span> res</span>
<span id="cb32-87"><a href="#cb32-87" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-88"><a href="#cb32-88" aria-hidden="true" tabindex="-1"></a><span class="kw">def</span> print_metrics(name, ref, pred):</span>
<span id="cb32-89"><a href="#cb32-89" aria-hidden="true" tabindex="-1"></a>  res <span class="op">=</span> build_metrics(ref, pred)</span>
<span id="cb32-90"><a href="#cb32-90" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">=</span> <span class="st">&#39;</span><span class="sc">{:15s}</span><span class="st">&#39;</span>.<span class="bu">format</span>(name)</span>
<span id="cb32-91"><a href="#cb32-91" aria-hidden="true" tabindex="-1"></a>  _acc <span class="op">=</span> <span class="st">&#39;</span><span class="sc">{:.1f}</span><span class="st">%&#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;acc&#39;</span>] <span class="op">*</span> <span class="fl">100.0</span>)</span>
<span id="cb32-92"><a href="#cb32-92" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:10s}</span><span class="st">&#39;</span>.<span class="bu">format</span>(_acc)</span>
<span id="cb32-93"><a href="#cb32-93" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;rmse&#39;</span>])</span>
<span id="cb32-94"><a href="#cb32-94" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;mae&#39;</span>])</span>
<span id="cb32-95"><a href="#cb32-95" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;var&#39;</span>])</span>
<span id="cb32-96"><a href="#cb32-96" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.6f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;f1_score&#39;</span>])</span>
<span id="cb32-97"><a href="#cb32-97" aria-hidden="true" tabindex="-1"></a>  <span class="bu">str</span> <span class="op">+=</span> <span class="st">&#39;</span><span class="sc">{:.8f}</span><span class="st">  &#39;</span>.<span class="bu">format</span>(res[<span class="st">&#39;l2r&#39;</span>])</span>
<span id="cb32-98"><a href="#cb32-98" aria-hidden="true" tabindex="-1"></a>  <span class="bu">print</span>(<span class="bu">str</span>)</span>
<span id="cb32-99"><a href="#cb32-99" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-100"><a href="#cb32-100" aria-hidden="true" tabindex="-1"></a><span class="co"># Reshape the outputs to be aligned</span></span>
<span id="cb32-101"><a href="#cb32-101" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-102"><a href="#cb32-102" aria-hidden="true" tabindex="-1"></a>p_ <span class="op">=</span> p_.reshape(r_ref.shape)</span>
<span id="cb32-103"><a href="#cb32-103" aria-hidden="true" tabindex="-1"></a>pm_ <span class="op">=</span> p_.reshape(r_ref.shape)</span>
<span id="cb32-104"><a href="#cb32-104" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-105"><a href="#cb32-105" aria-hidden="true" tabindex="-1"></a><span class="co"># Log the results</span></span>
<span id="cb32-106"><a href="#cb32-106" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-107"><a href="#cb32-107" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;</span><span class="ch">\n</span><span class="st">Evaluation report&#39;</span>)</span>
<span id="cb32-108"><a href="#cb32-108" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;-&#39;</span><span class="op">*</span><span class="dv">80</span>)</span>
<span id="cb32-109"><a href="#cb32-109" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;               </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">  </span><span class="sc">{:8s}</span><span class="st">&#39;</span>.<span class="bu">format</span>(<span class="st">&#39;acc&#39;</span>, <span class="st">&#39;rmse&#39;</span>,</span>
<span id="cb32-110"><a href="#cb32-110" aria-hidden="true" tabindex="-1"></a>          <span class="st">&#39;mae&#39;</span>, <span class="st">&#39;var&#39;</span>, <span class="st">&#39;f1_score&#39;</span>, <span class="st">&#39;l2r&#39;</span>))</span>
<span id="cb32-111"><a href="#cb32-111" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;-&#39;</span><span class="op">*</span><span class="dv">80</span>)</span>
<span id="cb32-112"><a href="#cb32-112" aria-hidden="true" tabindex="-1"></a>print_metrics(<span class="st">&#39;C-model&#39;</span>, r_ref, p_)</span>
<span id="cb32-113"><a href="#cb32-113" aria-hidden="true" tabindex="-1"></a>print_metrics(<span class="st">&#39;original model&#39;</span>, r_ref, pm_)</span>
<span id="cb32-114"><a href="#cb32-114" aria-hidden="true" tabindex="-1"></a>print_metrics(<span class="st">&#39;X-cross&#39;</span>, pm_, p_)</span>
<span id="cb32-115"><a href="#cb32-115" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;-&#39;</span><span class="op">*</span><span class="dv">80</span>)</span>
<span id="cb32-116"><a href="#cb32-116" aria-hidden="true" tabindex="-1"></a></span>
<span id="cb32-117"><a href="#cb32-117" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(<span class="st">&#39;&#39;</span>)</span>
<span id="cb32-118"><a href="#cb32-118" aria-hidden="true" tabindex="-1"></a>target_names <span class="op">=</span> [<span class="st">&#39;c0&#39;</span>, <span class="st">&#39;c1&#39;</span>, <span class="st">&#39;c2&#39;</span>, <span class="st">&#39;c3&#39;</span>, <span class="st">&#39;c4&#39;</span>, <span class="st">&#39;c5&#39;</span>, <span class="st">&#39;c6&#39;</span>, <span class="st">&#39;c7&#39;</span>, <span class="st">&#39;c8&#39;</span>, <span class="st">&#39;c9&#39;</span>]</span>
<span id="cb32-119"><a href="#cb32-119" aria-hidden="true" tabindex="-1"></a><span class="bu">print</span>(classification_report(np.argmax(r_ref, axis<span class="op">=</span><span class="dv">1</span>), np.argmax(p_, axis<span class="op">=</span><span class="dv">1</span>),</span>
<span id="cb32-120"><a href="#cb32-120" aria-hidden="true" tabindex="-1"></a>                            target_names<span class="op">=</span>target_names))</span></code></pre></div>
<!-- External ST resources/links -->
<!-- Internal resources/links -->
<!-- External resources/links -->
<!-- Cross references -->
</section>
<section id="references" class="level1">
<h1>References</h1>
<table>
<colgroup>
<col style="width: 18%" />
<col style="width: 81%" />
</colgroup>
<thead>
<tr class="header">
<th style="text-align: left;">ref</th>
<th style="text-align: left;">description</th>
</tr>
</thead>
<tbody>
<tr class="odd">
<td style="text-align: left;">[DS]</td>
<td style="text-align: left;">X-CUBE-AI - AI expansion pack for STM32CubeMX <a href="https://www.st.com/en/embedded-software/x-cube-ai.html">https://www.st.com/en/embedded-software/x-cube-ai.html</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[UM]</td>
<td style="text-align: left;">User manual - Getting started with X-CUBE-AI Expansion Package for Artificial Intelligence (AI) <a href="https://www.st.com/resource/en/user_manual/dm00570145.pdf">(pdf)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[CLI]</td>
<td style="text-align: left;">stm32ai - Command Line Interface <a href="command_line_interface.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[API]</td>
<td style="text-align: left;">Embedded inference client API <a href="embedded_client_api.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[METRIC]</td>
<td style="text-align: left;">Evaluation report and metrics <a href="evaluation_metrics.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[TFL]</td>
<td style="text-align: left;">TensorFlow Lite toolbox <a href="supported_ops_tflite.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[KERAS]</td>
<td style="text-align: left;">Keras toolbox <a href="supported_ops_keras.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[ONNX]</td>
<td style="text-align: left;">ONNX toolbox <a href="supported_ops_onnx.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[FAQS]</td>
<td style="text-align: left;">FAQ <a href="faq_generic.html">generic</a>, <a href="faq_validation.html">validation</a>, <a href="faq_quantization.html">quantization</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[QUANT]</td>
<td style="text-align: left;">Quantization and quantize command <a href="quantization.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[RELOC]</td>
<td style="text-align: left;">Relocatable binary network support <a href="relocatable.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[CUST]</td>
<td style="text-align: left;">Support of the Keras Lambda/custom layers <a href="keras_lambda_custom.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[TFLM]</td>
<td style="text-align: left;">TensorFlow Lite for Microcontroller support <a href="tflite_micro_support.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[INST]</td>
<td style="text-align: left;">Setting the environment <a href="setting_env.html">(link)</a></td>
</tr>
<tr class="odd">
<td style="text-align: left;">[OBS]</td>
<td style="text-align: left;">Platform Observer API <a href="api_platform_observer.html">(link)</a></td>
</tr>
<tr class="even">
<td style="text-align: left;">[C-RUN]</td>
<td style="text-align: left;">Executing locally a generated c-model <a href="how_to_run_a_model_locally.html">(link)</a></td>
</tr>
</tbody>
</table>
</section>



<section class="st_footer">

<h1> <br> </h1>

<p style="font-family:verdana; text-align:left;">
 Embedded Documentation 

	- <b> Evaluation report and metrics </b>
			<br> X-CUBE-AI Expansion Package
	 
			<br> r2.1
		 - AI PLATFORM r7.0.0
			 (Embedded Inference Client API 1.1.0) 
			 - Command Line Interface r1.5.1 
		
	
</p>

<img src="" title="ST logo" align="right" height="100" />

<div class="st_notice">
Information in this document is provided solely in connection with ST products.
The contents of this document are subject to change without prior notice.
<br>
© Copyright STMicroelectronics 2020. All rights reserved. <a href="http://www.st.com">www.st.com</a>
</div>

<hr size="1" />
</section>


</article>
</body>

</html>
